REVIEW 4 major objections 5 minor 36 references
MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Graph influence scores can identify which unlabeled examples are worth pseudo-labeling for many-shot in-context learning, letting LLMs run many-shot ICL with roughly 20 human labels.
desk verdict Useful empirical contribution with a broken proof and an overstated headline; worth refereeing, but the theory needs fixing or removal. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the node-influence lower bound of Theorem 3.2: over a graph with mean degree d, the expected log-influence from a node u to a node set V is at least the log of the geometric mean of shortest-path counts between u and V minus log d times the average shortest-path distance. MAPLE turns this bound into a ranking score for pseudo-label selection and a per-query ranking for demonstration selection. The graph is built from Contriever embeddings with edges to the k most relevant nodes, and the same construction is reused at query time with labels included in the embedding when available.
What would settle it
On one dataset with ground-truth labels held out, compute MAPLE's influence ranking, reveal the labels, and compare the pseudo-label accuracy of its top-100 selected samples with 100 randomly selected samples and with the top-100 by Contriever similarity; if the influence-selected set is not more accurate, or if substituting random pseudo-labels preserves performance, then the influence score is not doing the work the paper attributes to it.
Extended reading notes
Core claim
The central claim is that node influence on a k-nearest-neighbor graph built from Contriever query embeddings is a usable proxy for how valuable an unlabeled example will be as a pseudo-labeled demonstration. MAPLE defines the influence score s(V_L, v) = log f_PS(v, V_L) − log d · L_S(v, V_L), selects the top P unlabeled samples by this score, pseudo-labels them with an LLM, and then for each test query builds a fresh graph over the candidate pool plus the query, keeping the fraction α of candidates whose influence on the query node is highest. The paper reports that this consistently outperforms random and RAG/Contriever similarity baselines across eight datasets covering summarization, reasoning, classification, and question answering, with the largest gains on complex tasks, and that a stronger LLM amplifies the gains.
Load-bearing premise
The load-bearing premise is that an influence score computed on a k-nearest-neighbor graph of Contriever query embeddings ranks unlabeled samples in the same order as their actual usefulness as pseudo-labeled demonstrations for the downstream LLM.
Editorial extensions
If this is right
- With roughly 20 human labels, MAPLE can feed hundreds of LLM-pseudo-labeled demonstrations without the accuracy collapse seen when raw unlabeled demonstrations are used.
- Its per-query demonstration selection filters out unrelated pseudo-labeled examples, so accuracy grows with the pseudo-labeling budget on most tasks, while the paper also finds that on Tracking7 and Salient adding too many pseudo-labels can hurt.
- Using a stronger LLM for both pseudo-labeling and inference increases the gap over baselines, so the method's benefits scale with model capability.
- Dropping adaptive selection and caching the demonstration prefix with a KV cache trades a small amount of accuracy for lower inference time, and placing the few true labels nearest the query improves accuracy.
- Higher fractions of pseudo-labeled demonstrations can compensate for fewer human labels, though labeled demonstrations remain stronger guidance when available.
Reading between the lines
- A natural stress test is to replace the influence score with a simpler structural baseline, such as k-NN degree centrality or embedding density, and see whether the exact shortest-path counts are what carry the ranking or merely graph locality.
- Because the score uses only query embeddings, the same selection idea could be dropped into other label-hungry pipelines, such as active learning or instruction-data curation, where one must choose which unlabeled items to annotate.
- The paper's ranking guarantee rests on a lower bound applied to a k-NN graph; the empirical claim would be more firmly established by directly measuring whether top-ranked samples have higher pseudo-label accuracy than randomly chosen samples, and by checking sensitivity to k and to the embedding model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MAPLE, a framework for many-shot in-context learning when only a small labeled set and a large unlabeled pool are available. MAPLE first builds a k-NN graph over labeled and unlabeled queries using Contriever embeddings, then selects the top-P unlabeled nodes for pseudo-labeling according to an influence score s(V_L,v)=log f_PS(v,V_L)-log d * L_S(v,V_L) (Eq. 8), which is justified by Theorem 3.2 on node influence in a GCN. Pseudo-labels are generated by an LLM, and at inference MAPLE constructs a per-query graph over the candidate pool and selects a fixed fraction alpha of demonstrations by the same influence score (Eq. 14). Experiments on eight datasets with Gemini 1.5 Flash and Pro compare MAPLE against Zero-shot, Few-shot, Random, RAG, and RAG-Adapt under equal demonstration budgets, and the paper claims MAPLE consistently outperforms all baselines.
Significance. If the central claim holds, the paper addresses a practically important bottleneck of many-shot ICL: reducing reliance on expensive human labels by pseudo-labeling only a small number of well-chosen unlabeled examples. The empirical setup is macro-fair in that all methods receive the same labeled pool, the same pseudo-labeling budget, and the same test queries, and the paper includes several useful ablations (encoder choice, demonstration order, fraction of pseudo-labels, and the KV-cache trade-off). The release of code is a further strength. However, the theoretical support for the influence score appears to contain a genuine expectation/logarithm error, and the empirical evidence for the headline 'consistently outperforms' claim is weakened by the absence of significance tests and by several point differences that lie within one standard deviation. The significance of the paper will depend on whether the proof can be repaired or honestly downgraded to a heuristic and whether the ranking produced by the influence score can be directly validated against the LLM's downstream utility.
major comments (4)
- [Appendix A, Eqs. (21) and (24)] The proof of Lemma A.1 is not valid as written. Eq. (20) establishes a lower bound on E(||∂h_i/∂h_j||), i.e., E(I(v_i,v_j)), but Eq. (21) then treats E(log I(v_i,v_j)) as equal to log E(I(v_i,v_j)). Jensen's inequality gives the opposite direction, E(log I) ≤ log E(I), so the inequality in Eq. (21) does not follow. Eq. (24) repeats the same conflation when passing from E(log I) to sums of log E(I). Since Theorem 3.2 is used to justify the influence score in Eq. (8) and therefore the selection rules in Eqs. (9) and (14), this is a load-bearing error. The authors should either supply a correct proof, weaken the theorem to a heuristic statement, or remove the theorem as a claimed guarantee.
- [Sections 3.4 and 3.5] Even if the expectation/logarithm issue were repaired, Theorem 3.2 would at best bound log-influence in a linearized GCN with a random-degree assumption; it does not by itself establish that the Eq. (8) ranking matches the actual contribution of a pseudo-labeled sample when served as a demonstration to a Gemini-class LLM in many-shot ICL. The paper needs a direct validation of this transfer, for example by comparing the Eq. (8) ranking with an oracle ranking based on leave-one-out accuracy gains of candidate demonstrations on one or two datasets. Without such a test, the advantage of MAPLE over RAG-Adapt rests on an unverified proxy.
- [Table 1 and Section 4.2, Observation 1] The paper reports no significance tests, and several reported improvements over RAG are within one standard deviation of the baseline, e.g., Banking77 Gemini Flash at |D_U*|=20 (77.0±2.8 vs 77.0±2.3) and GPQA Gemini Pro at |D_U*|=100 (43.9±1.6 vs 43.3±1.0). Observation 3 of Section 4.2 admits limited benefits on Tracking7 and XSum, and Observation 5 admits performance drops on Tracking7 and Salient as pseudo-label count grows. The claim that MAPLE 'consistently outperforms all other baselines across all eight datasets' is therefore stronger than the evidence supports. Please add paired significance tests over the five runs or qualify the claim to specific datasets and demonstration budgets.
- [Section 3.5, Eq. (14)] The prose states that demonstrations are selected when their influence on the test node 'exceeds the average influence of labeled nodes,' but Eq. (14) instead defines a fixed-cardinality top-α selection by maximizing the sum of s(v,v_test). These two rules are different, and no algorithm is provided for the combinatorial argmax in Eq. (14). The exact selection rule and its implementation need to be stated precisely for reproducibility.
minor comments (5)
- [Throughout] There are several typos and naming inconsistencies: 'Contriver' should be 'Contriever', 'Data' should be 'Date' in Observation 2 of Section 4.2, 'Tracking77' should be 'Tracking7' in Observation 5, and 'KG Cache' should be 'KV Cache' in Section 4.5.
- [Table 3] The header for the reasoning/QA prompt reads 'Quesition Answering' and should be corrected to 'Question Answering.'
- [Definition 3.1] The notation I(v_i,v_j)=||∂v_i/∂v_j|| is imprecise: the derivatives are taken between node representations h_i and h_j, not between node objects themselves, and the specific norm used should be stated.
- [Introduction, Contributions] The paper claims to be 'the first to explore the capability of many-shot ICL under the pseudo-labeled setting,' but the Related Works section credits Agarwal et al. with using model-generated answers in many-shot ICL. The novelty claim should be reconciled with that discussion or narrowed to the selection and adaptation aspects.
- [Appendix A] The sentence following Eq. (19) says the expectation of node-degree products on a path is d^(n+1) under 'uniformly distributed' degrees; this requires an independence assumption on the path's node degrees, not merely a shared mean, and the assumption should be stated precisely.
Circularity Check
No significant circularity: the influence score is derived from a graph-theoretic lower bound and evaluated on held-out test sets; the Appendix A proof error is a correctness gap, not a circular reduction.
full rationale
MAPLE's two core mechanisms are the influence score s(VL,v)=log fPS(v,VL)-log d*LS(v,VL) (Eq. 8) for pseudo-label selection and the per-query demonstration selection score s(v,vtest) (Eq. 14). Both scores are produced by Theorem 3.2's lower bound on GNN node influence (Appendix A) using Contriever embeddings of the inputs and pseudo-labels; they are not fitted to test labels or to the final accuracy numbers. The reported accuracies are measured on held-out test splits against external baselines, so no predicted quantity is equivalent by construction to an input or fitted parameter. The paper's self-citations (Chen et al. 2024a, Wang et al. 2024a, Chen et al. 2025) appear in related work and do not carry the derivation of Eq. 8 or Eq. 14. One genuine concern is correctness rather than circularity: Appendix A asserts E(log I)=log E(I) in Eqs. (21) and (24), which violates Jensen's inequality, so the theorem's ranking guarantee is not established as written. That gap does not make the derivation circular, because the theorem is not the quantity being predicted and the score's ranking is not fitted to test outcomes.
Assumptions & free parameters
free parameters (5)
- k (graph neighborhood size) =
20
- alpha (adaptive selection fraction) =
0.75
- P (pseudo-labeling budget) =
20 to 100
- |D_L| labeled sample count =
20 (100 in Sec 4.3)
- mean degree d =
unstated
assumptions (4)
- ad hoc to paper Node degrees are randomly distributed with mean d.
- domain assumption Contriever query embeddings define a task-relevant similarity graph.
- domain assumption Pseudo-labels from the LLM are accurate enough to serve as demonstrations.
- domain assumption The lower bound in Theorem 3.2 is a valid ranking score.
Cite this review
Pith. "Pith review of MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning." pith.science (2026). https://pith.science/paper/7BK5QPUT
@misc{pith2026250516225,
author = {Pith},
title = {Pith review of: MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7BK5QPUT}},
note = {Machine review of arXiv:2505.16225}
}
read the original abstract
In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements in the expanded context windows of LLMs have led to many-shot ICL, which uses hundreds of demonstrations and outperforms few-shot ICL, which relies on fewer examples. However, this approach is often hindered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose Many-Shot Adaptive Pseudo-LabEling, namely MAPLE, a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information. We first identify a subset of impactful unlabeled samples and perform pseudo-labeling on them by querying LLMs. These pseudo-labeled samples are then adaptively selected and tailored to each test query as input to improve the performance of many-shot ICL, without significant labeling costs. Extensive experiments on real-world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance LLM adaptability and performance with limited labeled data.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
M., Bohnet, B., Rosias, L., Chan, S
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S. C., Zhang, B., Faust, A., and Larochelle, H. Many-shot in-context learning. In ICML 2024 Workshop on In-Context Learning,
work page 2024
-
[4]
Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long- context models: An in-depth exploration. arXiv preprint arXiv:2405.00200,
-
[5]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,
1901
-
[7]
FastGAS: Fast Graph-based Annotation Selection for In-Context Learning
Chen, Z., Wang, S., Shen, C., and Li, J. Fastgas: Fast graph- based annotation selection for in-context learning. arXiv preprint arXiv:2406.03730, 2024a. Chen, Z., Weiss, G., Mitchell, E., Celikyilmaz, A., and Bosselut, A. Reckoning: reasoning through dynamic knowledge encoding. Advances in Neural Information Processing Systems, 36, 2024b. Chen, Z., Wang,...
-
[10]
Multimodal task vectors enable many- shot multimodal in-context learning
Huang, B., Mitra, C., Arbelle, A., Karlinsky, L., Darrell, T., and Herzig, R. Multimodal task vectors enable many- shot multimodal in-context learning. arXiv preprint arXiv:2406.15334,
-
[11]
Unsupervised dense infor- mation retrieval with contrastive learning
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning. arXiv preprint arXiv:2112.09118,
-
[14]
S., Boratko, M., Luan, Y ., Arnold, S
Lee, J., Chen, A., Dai, Z., Dua, D., Sachan, D. S., Boratko, M., Luan, Y ., Arnold, S. M., Perot, V ., Dalmia, S., et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121,
-
[15]
In-context learning with many demonstration examples
Li, M., Gong, S., Feng, J., Xu, Y ., Zhang, J., Wu, Z., and Kong, L. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931,
Show all 36 references
-
[16]
D., Yue, X., and Chen, W
Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long- context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060,
-
[17]
What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804,
Liu, J., Shen, D., Zhang, Y ., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804,
-
[18]
Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity
Lu, Y ., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786,
-
[20]
In- context learning with iterative demonstration selection
Qin, C., Zhang, A., Chen, C., Dagar, A., and Ye, W. In- context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881,
-
[21]
and Gurevych, I
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084,
1908 arXiv
-
[23]
Learning to retrieve prompts for in-context learning
Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 2655–2671,
2022
-
[24]
H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N
Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N. A., et al. Selective annotation makes language models bet- ter few-shot learners. arXiv preprint arXiv:2209.01975,
-
[25]
W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. Challenging big-bench tasks and whether chain-of- thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051,
2023
-
[26]
I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al
Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530,
-
[28]
Learning to retrieve in-context examples for large language models
Wang, L., Yang, N., and Wei, F. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164,
-
[29]
and Liu, L
Wei, X. and Liu, L. Are large language models good in- context learners for financial sentiment analysis? arXiv preprint arXiv:2503.04873,
-
[30]
More is not always better? enhanc- ing many-shot in-context learning with differentiated and reweighting objectives
Zhang, X., Lv, A., Liu, Y ., Sung, F., Liu, W., Shang, S., Chen, X., and Yan, R. More is not always better? enhanc- ing many-shot in-context learning with differentiated and reweighting objectives. arXiv preprint arXiv:2501.04070,
-
[31]
X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223,
-
[32]
and (Xu et al., 2018), utilizing GCNs (Kipf & Welling,
2018
-
[33]
and GraphSAGE (Hamilton et al., 2017)) by assigning different values to edge weights. Specifically, the propagation process in thel-th layer can be represented as H(l+1) = σ( ˆAH(l)W(l), where H(l) and W(l) denote the node representation and weight parameter matrices, respecti...
2017
-
[34]
(2018), we set σ as the identity function and W as the identity matrix
Following the convention of Huang & Zitnik (2020), Wang & Leskovec (2020), and Xu et al. (2018), we set σ as the identity function and W as the identity matrix. Additionally, we assume that the propagation process is performed over a sufficient number of iterations. Consequent...
2020
-
[35]
What is the article about?
14 MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning Table 3: A list of prompts that we use for many-shot ICL on summarization, reasoning, and question answering tasks. Types Prompts Summarization You are an expert in article summarization. I am going to give y...
2018
-
[1937]
Sentence: Pharmaceuticals group Orion Corp reported a fall in its third-quarter earnings, which were impacted by larger expenditures on R&D and marketing. Label: negative
What is the date tomorrow in MM/DD/YYYY? Options: (A) 12/11/1937; (B) 12/25/1937; (C) 01/04/1938; (D) 12/04/1937; (E) 12/25/2006; (F) 07/25/1937" • Salient (Suzgun et al., 2023): This dataset is designed for Salient Translation Error Detection, where, given a source sentence w...
1937
-
[2014]
B., and Lapata, M
Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807,
2018
-
[2017]
Deberta: Decoding- enhanced bert with disentangled attention
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654,
2006 arXiv
-
[2018]
and Leskovec, J
Wang, H. and Leskovec, J. Unifying graph con- volutional neural networks and label propagation. arXiv:2002.06755,
2002 arXiv
-
[2019]
L., Stickland, A
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022,
-
[2020]
Efficient intent detection with dual sentence encoders
Casanueva, I., Temˇcinas, T., Gerz, D., Henderson, M., and Vuli´c, I. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807,
2003 arXiv
-
[2021]
M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C
Jain, S., Keshava, V ., Sathyendra, S. M., Fernandes, P., Liu, P., Neubig, G., and Zhou, C. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.01200,
-
[2022]
J., Gupta, P., Dalmia, S., Kolhar, P., et al
Baek, J., Lee, S. J., Gupta, P., Dalmia, S., Kolhar, P., et al. Revisiting in-context learning with long context language models. arXiv preprint arXiv:2412.16926,
-
[2023]
A., Wang, J
Jiang, Y ., Irvin, J. A., Wang, J. H., Chaudhry, M. A., Chen, J. H., and Ng, A. Y . Many-shot in-context learning in multimodal foundation models. In ICML 2024 Workshop on In-Context Learning,
2024
-
[2024]
In-context examples selection for machine translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., and Ghazvininejad, M. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437,
-
[2025]
Goemotions: A dataset of fine-grained emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547,
2005 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.