{"id":"5d42ff55-0b71-49f1-929e-c675e31c9d37","arxiv_id":"2505.16225","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.","lead":"A new framework, MAPLE, picks which unlabeled examples to label with an AI model and which demonstrations to feed to a large language model for each question. It improves many-shot in-context learning accuracy on several benchmarks while using only 20 human labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.2's proof conflates E(log I) with log E(I), so the influence-score ranking guarantee is unproven; without a positive correlation between the score and actual LLM utility, the 'consistently outperforms' claim is not established.","rationale":"The reader identified the same load-bearing weak point: the influence-score ranking is the premise that must hold for the central claim, and the Appendix proof contains log-expectation errors. I agree with that assessment. My concrete test goes beyond the proof issue by directly probing whether the proxy correlates with downstream utility; this is the condition that must hold for the central claim to survive. The empirical tables show many small, within-error differences, so a direct ranking test is the decisive check. I do not see a basis for rejecting the paper outright: the method is clearly specified, code is released, and the empirical protocol is mostly reproducible. The correct verdict remains CONDITIONAL, as the ranking justification and the significance of the improvements need to be repaired or substantiated. If the ranking test fails, the method may still work as a heuristic, but the paper's claimed theoretical grounding and the specific 'influence-based' novelty would need to be downgraded; if it passes and significance tests support Table 1, the central claim holds.","tokens_in":24895,"tokens_out":11224,"duration_ms":96035,"concrete_test":"On two datasets (e.g., Date and Banking77), compute the MAPLE influence score s(VL,v) for every candidate unlabeled sample using the paper's protocol (k=20, Contriever). For a fixed set of 20 labeled demonstrations and a fixed pseudo-labeling budget, measure each candidate's actual utility as the change in Gemini-1.5-Flash accuracy when that candidate (with its pseudo-label) is added to the demonstration pool for a held-out query set, averaging over queries and pseudo-label noise. Then compute Spearman's rank correlation between the influence score and the measured utility. If the correlation is not significantly positive (or is negative), the ranking guarantee underlying Eq. 8/14 is empirically unsupported, and the central claim that MAPLE's advantage comes from influence-based selection collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core mechanism of MAPLE is the influence score s(VL,v) = log fPS(v,VL) - log d * LS(v,VL) (Eq. 8), used both to select pseudo-label candidates and to choose per-query demonstrations (Eq. 14). The paper justifies this score with Theorem 3.2, but the proof is invalid: Eq. 21 asserts E(log I) = log E(I), and Eq. 24 repeats the same conflation, whereas Jensen's inequality gives E(log I) <= log E(I). The direction needed for a lower bound is therefore not established, so the theorem does not support the score as stated. Separately, even a correct version of the theorem would bound log-influence in a linearized GCN with random degrees (Appendix A, Eqs. 16-20); it says nothing about whether the resulting ranking matches the actual contribution of a pseudo-labeled sample to the Gemini model's many-shot ICL accuracy. The empirical headline ('consistently outperforms all baselines across all eight datasets', Sec. 4.2) thus rests on an unvalidated proxy: if the graph-theoretic score ranks samples differently from their real downstream utility, MAPLE's advantage over RAG-Adapt could disappear. Table 1 shows several entries where MAPLE and RAG differ by less than one standard deviation (e.g., Banking77 Flash at 20 pseudo-labeled demos: 77.0±2.8 vs 77.0±2.3; GPQA Pro at 100: 43.9±1.6 vs 43.3±1.0), so the claimed consistent superiority is not robust without a direct test of the ranking.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAPLE, a framework for many-shot in-context learning when only a small labeled set and a large unlabeled pool are available. MAPLE first builds a k-NN graph over labeled and unlabeled queries using Contriever embeddings, then selects the top-P unlabeled nodes for pseudo-labeling according to an influence score s(V_L,v)=log f_PS(v,V_L)-log d * L_S(v,V_L) (Eq. 8), which is justified by Theorem 3.2 on node influence in a GCN. Pseudo-labels are generated by an LLM, and at inference MAPLE constructs a per-query graph over the candidate pool and selects a fixed fraction alpha of demonstrations by the same influence score (Eq. 14). Experiments on eight datasets with Gemini 1.5 Flash and Pro compare MAPLE against Zero-shot, Few-shot, Random, RAG, and RAG-Adapt under equal demonstration budgets, and the paper claims MAPLE consistently outperforms all baselines.","tokens_in":25166,"tokens_out":5031,"duration_ms":42782,"significance":"If the central claim holds, the paper addresses a practically important bottleneck of many-shot ICL: reducing reliance on expensive human labels by pseudo-labeling only a small number of well-chosen unlabeled examples. The empirical setup is macro-fair in that all methods receive the same labeled pool, the same pseudo-labeling budget, and the same test queries, and the paper includes several useful ablations (encoder choice, demonstration order, fraction of pseudo-labels, and the KV-cache trade-off). The release of code is a further strength. However, the theoretical support for the influence score appears to contain a genuine expectation/logarithm error, and the empirical evidence for the headline 'consistently outperforms' claim is weakened by the absence of significance tests and by several point differences that lie within one standard deviation. The significance of the paper will depend on whether the proof can be repaired or honestly downgraded to a heuristic and whether the ranking produced by the influence score can be directly validated against the LLM's downstream utility.","major_comments":[{"comment":"The proof of Lemma A.1 is not valid as written. Eq. (20) establishes a lower bound on E(||∂h_i/∂h_j||), i.e., E(I(v_i,v_j)), but Eq. (21) then treats E(log I(v_i,v_j)) as equal to log E(I(v_i,v_j)). Jensen's inequality gives the opposite direction, E(log I) ≤ log E(I), so the inequality in Eq. (21) does not follow. Eq. (24) repeats the same conflation when passing from E(log I) to sums of log E(I). Since Theorem 3.2 is used to justify the influence score in Eq. (8) and therefore the selection rules in Eqs. (9) and (14), this is a load-bearing error. The authors should either supply a correct proof, weaken the theorem to a heuristic statement, or remove the theorem as a claimed guarantee.","section":"Appendix A, Eqs. (21) and (24)"},{"comment":"Even if the expectation/logarithm issue were repaired, Theorem 3.2 would at best bound log-influence in a linearized GCN with a random-degree assumption; it does not by itself establish that the Eq. (8) ranking matches the actual contribution of a pseudo-labeled sample when served as a demonstration to a Gemini-class LLM in many-shot ICL. The paper needs a direct validation of this transfer, for example by comparing the Eq. (8) ranking with an oracle ranking based on leave-one-out accuracy gains of candidate demonstrations on one or two datasets. Without such a test, the advantage of MAPLE over RAG-Adapt rests on an unverified proxy.","section":"Sections 3.4 and 3.5"},{"comment":"The paper reports no significance tests, and several reported improvements over RAG are within one standard deviation of the baseline, e.g., Banking77 Gemini Flash at |D_U*|=20 (77.0±2.8 vs 77.0±2.3) and GPQA Gemini Pro at |D_U*|=100 (43.9±1.6 vs 43.3±1.0). Observation 3 of Section 4.2 admits limited benefits on Tracking7 and XSum, and Observation 5 admits performance drops on Tracking7 and Salient as pseudo-label count grows. The claim that MAPLE 'consistently outperforms all other baselines across all eight datasets' is therefore stronger than the evidence supports. Please add paired significance tests over the five runs or qualify the claim to specific datasets and demonstration budgets.","section":"Table 1 and Section 4.2, Observation 1"},{"comment":"The prose states that demonstrations are selected when their influence on the test node 'exceeds the average influence of labeled nodes,' but Eq. (14) instead defines a fixed-cardinality top-α selection by maximizing the sum of s(v,v_test). These two rules are different, and no algorithm is provided for the combinatorial argmax in Eq. (14). The exact selection rule and its implementation need to be stated precisely for reproducibility.","section":"Section 3.5, Eq. (14)"}],"minor_comments":[{"comment":"There are several typos and naming inconsistencies: 'Contriver' should be 'Contriever', 'Data' should be 'Date' in Observation 2 of Section 4.2, 'Tracking77' should be 'Tracking7' in Observation 5, and 'KG Cache' should be 'KV Cache' in Section 4.5.","section":"Throughout"},{"comment":"The header for the reasoning/QA prompt reads 'Quesition Answering' and should be corrected to 'Question Answering.'","section":"Table 3"},{"comment":"The notation I(v_i,v_j)=||∂v_i/∂v_j|| is imprecise: the derivatives are taken between node representations h_i and h_j, not between node objects themselves, and the specific norm used should be stated.","section":"Definition 3.1"},{"comment":"The paper claims to be 'the first to explore the capability of many-shot ICL under the pseudo-labeled setting,' but the Related Works section credits Agarwal et al. with using model-generated answers in many-shot ICL. The novelty claim should be reconciled with that discussion or narrowed to the selection and adaptation aspects.","section":"Introduction, Contributions"},{"comment":"The sentence following Eq. (19) says the expectation of node-degree products on a path is d^(n+1) under 'uniformly distributed' degrees; this requires an independence assumption on the path's node degrees, not merely a shared mean, and the assumption should be stated precisely.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The main technical problem is confined to the proof of Theorem 3.2 and the strength of the empirical claim. I do not see an irreparable circularity or an invented result: the method is evaluated on external benchmarks and the score is derived from embeddings and graph structure. The most constructive path for the authors is to downgrade Theorem 3.2 to a heuristic motivation, add a direct ranking-oracle experiment, and rephrase the 'consistently outperforms' claim with significance tests or dataset-specific qualifications. If the proof cannot be corrected, the paper should not present Eq. (7) as a theorem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MAPLE is a useful engineering contribution with a broken proof and an overstated headline. The practical idea — select unlabeled samples for pseudo-labeling via an influence score on a k-NN graph, then adaptively pick demonstrations per query — is new in many-shot ICL as far as I can tell, and the paper ships code, prompts, and most datasets. The experiments are mostly fair: same demonstration budgets and same pseudo-labeling budgets across baselines, eight datasets, two Gemini models, and useful ablations (encoder choice, demo order, label fraction, KV-cache). The gains over RAG-Adapt are typically 1–3% absolute, which is modest but believable for cutting labeling effort from hundreds to 20–100 manual labels.\n\nThe soft spots are real. Theorem 3.2 is not proven as written: Appendix A's Eq. 21 and Eq. 24 both assume E(log I) = log E(I), but Jensen's inequality gives E(log I) ≤ log E(I), the wrong direction for a lower bound. That matters because the theorem is the stated justification for the selection score. Even a corrected GCN-influence bound would only say something about the graph proxy, not about whether the ranking transfers to the LLM's actual behavior. The empirical results are the real evidence, and they are plausible, but the paper gives no significance tests, and Table 1 has entries where MAPLE and RAG differ by less than one standard deviation (e.g., Banking77 Flash at 20 demos: 77.0 vs 77.0; GPQA Pro at 100: 43.9 vs 43.3). The paper's own observations 3 and 5 admit limited or negative gains on Tracking7 and XSum, so the abstract's \"consistently outperforms\" is too strong. Also, the novelty claim \"first to explore pseudo-labeled many-shot ICL\" conflicts with the Related Work's own description of Agarwal et al. (2024), which used model-generated pseudo-labels without selection.\n\nMinor but worth noting: the setup underspecifies how D_L and D_U are sampled and how the mean degree d is estimated. These are easy fixes.\n\nWho should read this: anyone doing many-shot ICL or demonstration selection with scarce labels. It deserves a serious referee — the method is simple enough to be reproducible and the empirical trend is coherent — but the theorem section should be fixed or removed, and the statistical claims need qualification.\n\nRecommendation: send to peer review, with a heavy-revision mindset. The core idea survives contact with the data; the theory doesn't survive contact with Jensen.","headline":"Useful empirical contribution with a broken proof and an overstated headline; worth refereeing, but the theory needs fixing or removal.","tokens_in":25790,"tokens_out":4317,"would_cite":true,"duration_ms":35652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph influence scores can identify which unlabeled examples are worth pseudo-labeling for many-shot in-context learning, letting LLMs run many-shot ICL with roughly 20 human labels.","keywords":["in-context learning","many-shot ICL","pseudo-labeling","node influence","graph-based sample selection","adaptive demonstration selection","large language models","limited labeled data"],"falsifier":"On one dataset with ground-truth labels held out, compute MAPLE's influence ranking, reveal the labels, and compare the pseudo-label accuracy of its top-100 selected samples with 100 randomly selected samples and with the top-100 by Contriever similarity; if the influence-selected set is not more accurate, or if substituting random pseudo-labels preserves performance, then the influence score is not doing the work the paper attributes to it.","tokens_in":24600,"feed_emoji":"🎯","tokens_out":5459,"duration_ms":44962,"temperature":0.7,"pith_summary":"Many-shot in-context learning (hundreds of demonstrations) beats few-shot learning, but it normally needs many labeled examples. MAPLE replaces most of those labels with pseudo-labels produced by the LLM itself, and it tries to choose which unlabeled samples to label and which demonstrations to show for each query. The paper claims that a graph-theoretic influence score, based on the number of shortest paths from an unlabeled sample to the labeled set discounted by path length, ranks samples better than random or similarity-based selection. If this is right, MAPLE lets practitioners get many-shot ICL quality with only a small number of human labels, which matters whenever annotation is expensive.","feed_headline":"A graph influence score picks which unlabeled examples to pseudo-label","feed_subtitle":"Ranks unlabeled samples by graph influence and tailors demonstrations per query, beating baselines on eight datasets.","key_machinery":"The machinery is the node-influence lower bound of Theorem 3.2: over a graph with mean degree d, the expected log-influence from a node u to a node set V is at least the log of the geometric mean of shortest-path counts between u and V minus log d times the average shortest-path distance. MAPLE turns this bound into a ranking score for pseudo-label selection and a per-query ranking for demonstration selection. The graph is built from Contriever embeddings with edges to the k most relevant nodes, and the same construction is reused at query time with labels included in the embedding when available.","core_discovery":"The central claim is that node influence on a k-nearest-neighbor graph built from Contriever query embeddings is a usable proxy for how valuable an unlabeled example will be as a pseudo-labeled demonstration. MAPLE defines the influence score s(V_L, v) = log f_PS(v, V_L) − log d · L_S(v, V_L), selects the top P unlabeled samples by this score, pseudo-labels them with an LLM, and then for each test query builds a fresh graph over the candidate pool plus the query, keeping the fraction α of candidates whose influence on the query node is highest. The paper reports that this consistently outperforms random and RAG/Contriever similarity baselines across eight datasets covering summarization, reasoning, classification, and question answering, with the largest gains on complex tasks, and that a stronger LLM amplifies the gains.","pith_inferences":["A natural stress test is to replace the influence score with a simpler structural baseline, such as k-NN degree centrality or embedding density, and see whether the exact shortest-path counts are what carry the ranking or merely graph locality.","Because the score uses only query embeddings, the same selection idea could be dropped into other label-hungry pipelines, such as active learning or instruction-data curation, where one must choose which unlabeled items to annotate.","The paper's ranking guarantee rests on a lower bound applied to a k-NN graph; the empirical claim would be more firmly established by directly measuring whether top-ranked samples have higher pseudo-label accuracy than randomly chosen samples, and by checking sensitivity to k and to the embedding model."],"forward_implications":["With roughly 20 human labels, MAPLE can feed hundreds of LLM-pseudo-labeled demonstrations without the accuracy collapse seen when raw unlabeled demonstrations are used.","Its per-query demonstration selection filters out unrelated pseudo-labeled examples, so accuracy grows with the pseudo-labeling budget on most tasks, while the paper also finds that on Tracking7 and Salient adding too many pseudo-labels can hurt.","Using a stronger LLM for both pseudo-labeling and inference increases the gap over baselines, so the method's benefits scale with model capability.","Dropping adaptive selection and caching the demonstration prefix with a KV cache trades a small amount of accuracy for lower inference time, and placing the few true labels nearest the query improves accuracy.","Higher fractions of pseudo-labeled demonstrations can compensate for fewer human labels, though labeled demonstrations remain stronger guidance when available."],"supporting_citations":[{"why":"It supplies the Contriever encoder used to build the k-NN graph and compute relevance scores.","marker":"Izacard et al., 2021"},{"why":"It establishes the many-shot ICL setting with hundreds of demonstrations that MAPLE extends with pseudo-labeling.","marker":"Agarwal et al., 2024"},{"why":"It supplies the node-influence lower-bound technique and proof pattern that Theorem 3.2 builds on.","marker":"Huang & Zitnik, 2020"},{"why":"It provides the node-influence definition used in Definition 3.1.","marker":"Xu et al., 2018"},{"why":"It provides the Gemini 1.5 Flash and Gemini 1.5 Pro models used for pseudo-labeling and inference.","marker":"Team et al., 2024"},{"why":"It provides the BBH datasets Date, Salient, and Tracking7 used in the evaluation.","marker":"Suzgun et al., 2023"},{"why":"It provides the GPQA question-answering dataset used in the evaluation.","marker":"Rein et al., 2023"}],"fun_headline_variants":["Graph influence selects pseudo-labels for many-shot ICL","Influence-based pseudo-labeling for adaptive many-shot ICL","MAPLE: influence picks pseudo-labels for cheaper ICL","Pseudo-labels chosen by graph influence cut labeling costs","Adaptive pseudo-labeling via graph influence improves ICL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an influence score computed on a k-nearest-neighbor graph of Contriever query embeddings ranks unlabeled samples in the same order as their actual usefulness as pseudo-labeled demonstrations for the downstream LLM.","fun_headline_variants_meta":{"raw":{"variants":["Graph influence selects pseudo-labels for many-shot ICL","Influence-based pseudo-labeling for adaptive many-shot ICL","MAPLE: influence picks pseudo-labels for cheaper ICL","Pseudo-labels chosen by graph influence cut labeling costs","Adaptive pseudo-labeling via graph influence improves ICL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4318,"prompt_tokens":918,"completion_tokens":3400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":3317}},"tokens_in":534,"tokens_out":3400,"duration_ms":21063,"temperature":1.0,"reasoning_tokens":3317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:05:01.359035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On one dataset with ground-truth labels held out, compute MAPLE's influence ranking, reveal the labels, and compare the pseudo-label accuracy of its top-100 selected samples with 100 randomly selected samples and with the top-100 by Contriever similarity; if the influence-selected set is not more accurate, or if substituting random pseudo-labels preserves performance, then the influence score is not doing the work the paper attributes to it.","supporting_citations":[{"cited_title":"M., Bohnet, B., Rosias, L., Chan, S","cited_arxiv_id":null,"evidence_quote":"It establishes the many-shot ICL setting with hundreds of demonstrations that MAPLE extends with pseudo-labeling."},{"cited_title":"(2018), we set σ as the identity function and W as the identity matrix","cited_arxiv_id":null,"evidence_quote":"It supplies the node-influence lower-bound technique and proof pattern that Theorem 3.2 builds on."},{"cited_title":"W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al","cited_arxiv_id":null,"evidence_quote":"It provides the BBH datasets Date, Salient, and Tracking7 used in the evaluation."}],"review_version":1}