Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval

Fausto Giunchiglia; Hao Xu; Rite Bo; Rui Song; Yingji Li

REVIEW 4 major objections 6 minor 2 references

Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An OOD proxy from a source-tuned LLM and its unmodified base model can rank better in-context demonstrations without any target-domain access.

desk verdict Practical target-free demo retrieval that works on BOSS; the proxy ranking story is empirically useful but the theory only bounds ratio error, not ranking fidelity. read the letter →

arxiv 2606.00014 v1 pith:O2WLUGGW submitted 2026-04-13 cs.CL cs.AI

Hao Xu , Rite Bo , Fausto Giunchiglia , Yingji Li , Rui Song This is my paper

classification cs.CLcs.AI

keywords in-contextlearningout-of-distributiondemonstrationretrievalOODproxyMahalanobisdiversitylargelanguagemodelsdistributionshift

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models lose accuracy when the test domain differs from the examples they are shown. The usual remedy—retrieve similar source examples—breaks when the target domain is completely unavailable, so no one can measure similarity to it. This paper argues that a cheap stand-in works: instruction-tune a copy of the model on the source domain, leave another copy untouched, and score every source example by the log-perplexity gap between the two. Low-gap examples are treated as more target-like; a Mahalanobis diversity step then keeps the final shortlist varied. Across multiple open and closed models and four BOSS tasks, the resulting demonstrations raise out-of-distribution in-context accuracy over random, nearest-neighbor, influence, rewrite, and diversity baselines. Anyone who must ship LLMs into new domains without labeled target data has a concrete, target-free retrieval recipe.

What carries the argument

The OOD proxy score S(x) = log PPL of the unmodified base model minus log PPL of the source-tuned model (Eq. 3), whose error relative to the true likelihood ratio is bounded by Theorem 1 under mild KL and pointwise-mass assumptions; the score first filters a candidate pool, after which similarity plus a global Mahalanobis diversity constraint (Algorithm 1) selects the final demonstrations.

What would settle it

Construct a target domain known to lie far outside the base model’s pretraining support (e.g., highly specialized medical notes or synthetic adversarial NLI) and check whether the proxy ranking still beats random and pure KNN selection by a statistically significant margin on held-out accuracy; failure on that regime would falsify the claim that the base model is a reliable target proxy.

Watch

Extended reading notes

Core claim

DOPA shows that the log-perplexity difference between a source-domain instruction-tuned LLM and the same unmodified base LLM is a usable ranking signal for source examples that behave like an inaccessible target domain, and that combining this ranking with a Mahalanobis-distance diversity constraint yields demonstrations that improve OOD in-context learning across classification and NER tasks on multiple model families.

Load-bearing premise

The method assumes that an unmodified pretrained language model is already a good enough weak stand-in for whatever unknown target domain the user will later face.

Editorial extensions

If this is right

Practitioners can improve OOD in-context learning without collecting any target-domain examples.
A single lightweight LoRA instruction-tuning pass on source data is enough to build the ranking signal.
Pure semantic nearest-neighbor retrieval can underperform or even hurt under strong shift; a behavioral proxy is needed.
Mahalanobis diversity keeps the proxy from collapsing onto short, high-frequency patterns.
Closed-source models can still benefit by using an open model as the proxy, as the paper already shows for GPT-4o-mini and GPT-3.5-turbo.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the base model’s pretraining already covers the target well, the proxy signal strengthens; if the target is truly novel jargon or adversarial text, ranking may degrade toward random.
The same adapted-versus-unadapted pair can serve as a cheap density-ratio estimator for any retrieval or filtering problem where the target distribution is unobserved or private.
Replacing the single base-model proxy with an ensemble or a continually updated prior would be a direct way to tighten the practical error of Theorem 1.
The method’s success on NER and extractive QA hints that the same proxy may help token-level and span-level selection, not only sentence-level classification.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: DOPA is a usable pipeline for picking source demonstrations when you never see the target domain—source-tuned LLM vs base LLM log-perplexity ratio, then similarity plus Mahalanobis diversity. Gains on BOSS are consistent across models; the math is clean but thinner than the ranking claim needs.

What is new is the packaging, not the ingredients. Likelihood-ratio OOD scoring and embedding retrieval are known; the contribution is the concrete source-proxy / base-proxy construction for inaccessible-target ICL, the diversity step, and a multi-model evaluation that actually stresses OOD. Theorem 1 is a standard KL-plus-mass lower-bound guarantee on the log-ratio error; Lemma 1 correctly shows a uniform target proxy is looser. That is real formal work, not decoration.

The experiments are the paper’s strength. Six open models, two closed APIs, SA/TD/NLI/NER (plus a short QA check), ablations of proxy/similarity/Mahalanobis/uniform, k and N sweeps, Wilcoxon tests, energy KDE and distance plots, and a code link. Ablations line up with the story: drop similarity and you hurt most; uniform proxy underperforms the LLM proxy. Limitations are stated plainly.

Soft spots, in proportion. The stress-test is right on one point: Theorem 1 does not prove that S(x) preserves the true target-alignment ranking, only that the proxy log-ratio stays close if ε and m behave. If the base prior is badly off the true target, the bound can be vacuous and ˆD_S may just prefer “less source-like” samples. For closed-source models they substitute Llama-3.2-3B as both proxies, so the ranking signal is decoupled from the model that does ICL. Gains are often a few points, and k/N are chosen on the evaluation suite. None of that is a load-bearing collapse; it means the central assumption is empirical and should be treated as such in revision.

This is for people who do robust ICL or demo selection under domain shift. It will not reorganize the field, but it is a serious methods paper with inspectable claims. I would send it to peer review and engage with it if I were working in that lane.

Referee Report

4 major / 6 minor

Summary. The paper proposes DOPA, a target-inaccessible demonstration retrieval framework for OOD in-context learning. It constructs an OOD proxy from a source-instruction-tuned LLM and the unmodified base LLM, scores source samples by the log-perplexity difference S(x) (Eq. 3), filters a candidate pool, and then retrieves demonstrations by combining representation similarity with a Mahalanobis-distance diversity constraint (Algorithm 1). Theorem 1 bounds the proxy log-ratio error under KL and pointwise lower-bound assumptions; Lemma 1 argues that a uniform target proxy yields a looser bound. Experiments on the BOSS benchmark (SA, TD, NLI, NER) across six open-source and two closed-source LLMs report consistent gains over Random, KNN, DrICL, Rewrite, InfICL, and DICL, with ablations, k/N sweeps, Wilcoxon tests, and energy-score visualizations supporting the design.

Significance. If the empirical gains hold under stricter scrutiny of the proxy, the work addresses a practically important setting: demonstration selection when no target-domain pool is available. The multi-model, multi-task evaluation, component ablations, significance testing, released code, and energy-score analyses are concrete strengths. The proxy idea is a useful bridge between OOD detection and ICL retrieval. The theoretical contribution is modest but clean as a ratio-error bound; its main value is framing rather than a ranking guarantee. Overall significance is solid for the ICL/OOD robustness literature, contingent on clearer limits of the base-LLM target proxy.

major comments (4)

[§3.2 Theorem 1 / Eq. (3)] §3.2, Theorem 1 and Eq. (3): Theorem 1 bounds |log(P_t/P_s) − log(P^p_t/P^p_s)| under KL and m_t, m_s > 0, but does not establish that the ranking induced by S(x) preserves the ranking of the true likelihood ratio, nor that ε_t is small for the unmodified base LLM on BOSS targets. Contribution (i) and the abstract present this as demonstrating proxy “soundness” for selecting target-aligned samples. Please either (a) state the claim strictly as a ratio-error bound and separate it from ranking claims, or (b) add analysis (e.g., rank correlation / retrieval metrics against an oracle with target access) showing when S(x) preserves useful orderings. Without this, the theory does not underwrite the central retrieval claim.
[§3.2 / Appendix F] §3.2 (“LLMs can act as weak proxies for the target distribution”) and Appendix F (anli KDE): The method’s load-bearing assumption is that the unmodified base model keeps the target proxy close enough that low S(x) means “more target-like,” not merely “less source-like.” The paper itself notes that on anli the proxy distribution fails to leave the source. Please quantify proxy quality more systematically (e.g., energy/KL overlap of ˆD_S vs D_T across all datasets; failure cases; correlation of S(x) with target-domain energy or accuracy contribution) and discuss when the assumption breaks. Lemma 1 only rules out the uniform proxy; it does not calibrate the base-LLM prior.
[Appendix B.3 / Tables 2–3] Appendix B.3 and Tables 2–3: For GPT-4o-mini and GPT-3.5-turbo, both source and target proxies are replaced by Llama-3.2-3B, so S(x) is fully decoupled from the model that performs ICL. Gains on closed-source models are among the more interesting results, yet this substitution is only briefly noted. Please justify why a third-party proxy still ranks useful demonstrations for a different model family, report sensitivity when the proxy model changes (partially in Table 5 for open models), and avoid implying that the same proxy construction used for open models applies unchanged to closed APIs.
[Tables 1–3] Tables 1–3: Several reported gains over strong baselines are small or mixed (e.g., NLI averages often within ~1 point; some cells where Random or InfICL match or beat DOPA). Wilcoxon significance over nine tasks is helpful, but the OOD-robustness claim would be clearer with effect-size reporting, confidence intervals or multi-seed variance for non-deterministic baselines, and explicit discussion of when DOPA does not help. This is needed so readers can judge practical impact rather than average rank alone.

minor comments (6)

[§3.2 Eqs. (2)–(3)] Eq. (2) writes S(x) = P_target/P_source ≈ proxy ratio, while Eq. (3) switches to log-PPL difference. State explicitly that lower S(x) is preferred and that log-PPL difference is monotone with the log-likelihood form used in Theorem 1.
[Algorithm 1] Algorithm 1: notation is slightly inconsistent (C as both initial candidate size and a counter index; “Div Ddemo ≤ Div {ˆx}∪Ddemo”). Clarify initialization of C and the acceptance rule.
[Figure 1 / §3.1] Figure 1 caption and §3.1: “N×|Y| labeled examples” vs Algorithm 1 “size N” — unify demonstration-count notation for classification vs generative tasks.
[§2] Related work: briefly contrast with other target-free or domain-generalization ICL selection methods beyond the listed baselines; a short positioning paragraph would help.
[Throughout] Typos/style: “ademonstrationoptimization”, “OODproxyassessment”, “Wkeglarczyk” (Węglarczyk), and occasional missing spaces after periods in the PDF source.
[Appendix B.1] Appendix B.1: excluding CoNLL for annotation errors is reasonable; state how label-space unification for NER (Location/Organization/Person only) affects comparability to prior BOSS numbers.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: proxy score and retrieval are constructive, theory is a standard KL bound independent of results, and claims are evaluated on held-out targets.

full rationale

The core construction (source proxy via instruction tuning on DS, target proxy as the unmodified base LLM, S(x) = log PPL_target_proxy(x) - log PPL_source_proxy(x) in Eq. 3, k-filter, then cosine + Mahalanobis diversity in Alg. 1) is definitional and algorithmic; it does not embed the downstream accuracy numbers. Theorem 1 is a generic bound |log(Pt/Ps) - log(Pp_t/Pp_s)| ≤ εt/mt + εs/ms derived from KL and pointwise lower bounds; the short proof in Appendix A uses only the triangle inequality and the definition of KL, with no reference to BOSS accuracies or fitted values. Lemma 1 simply specializes the same bound to a uniform proxy. Experiments compare DOPA against Random/KNN/DrICL/Rewrite/InfICL/DICL on held-out target splits of BOSS (never used to construct the proxies or the ranking), with Wilcoxon tests. The only mild empirical circularity is the usual one: k ∈ {300,500,800,1000} and N ∈ {1..5} are selected after inspecting accuracy curves on the same evaluation suite (Figs. 2–3, Sec. 4.3). That does not force the ranking or the accuracy gains by construction. Self-citations (Song et al. 2024a/b/2025) appear only in related-work or motivation and are not used to justify the proxy, the bound, or uniqueness. Closed-source experiments substitute Llama-3.2-3B as proxy (App. B.3), which is a practical decoupling, not a circular reduction. Overall the paper is self-contained against external baselines; score 1 only for the ordinary hyper-parameter look-ahead.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claim rests on a small set of modeling choices (base LLM as target proxy, KL-bounded proxies, fixed k/N) plus standard mathematical inequalities. No new physical entities are postulated; the free parameters are ordinary hyper-parameters of the retrieval pipeline.

free parameters (4)

k (proxy-filtered subset size) = 800
Chosen as 800 after a sweep on the evaluation tasks; directly controls which source samples enter the candidate pool.
N (demonstrations per label) = 3
Fixed at 3 after an N-sweep; determines the final prompt size and therefore the measured accuracy.
C (initial candidate set size) = |Y| or 1
Set to |Y| for classification and 1 for generation; seeds the diversity-constrained search.
LoRA fine-tuning hyper-parameters = lr=1e-5, 1 epoch
Learning rate 1e-5, one epoch; define the source proxy and therefore the OOD scores.

assumptions (3)

domain assumption The unmodified base LLM is a sufficiently accurate weak proxy for the unknown target distribution.
Stated in Section 3.2 and used to justify replacing P_target with the base model; if false the ranking signal collapses.
domain assumption KL(P_target || P_proxy_target) ≤ ε_t, KL(P_source || P_proxy_source) ≤ ε_s and pointwise lower bounds m_t, m_s > 0 hold for the relevant samples.
Assumptions of Theorem 1 that convert KL bounds into a uniform log-ratio error bound; not verified empirically for the concrete LLMs.
standard math Triangle inequality and the definition of KL divergence yield the stated proxy-error bound.
Standard information-theoretic steps used in the Appendix A proof.

invented entities (2)

OOD proxy (source-proxy / target-proxy pair)
purpose: Approximate the inaccessible target distribution so that source samples can be ranked by estimated target-alignment.
Constructed ad hoc from an instruction-tuned and an untuned copy of the same LLM; independent evidence is only the downstream accuracy gains and energy-score visualizations, not an external falsifiable prediction.
S(x) = log PPL_target_proxy(x) − log PPL_source_proxy(x)
purpose: Scalar OOD score used to filter the source pool before similarity and diversity stages.
Direct operationalization of the likelihood-ratio idea for autoregressive LLMs; its validity inherits the proxy assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval." pith.science (2026). https://pith.science/paper/O2WLUGGW

@misc{pith2026260600014,
  author       = {Pith},
  title        = {Pith review of: Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2WLUGGW}},
  note         = {Machine review of arXiv:2606.00014}
}

read the original abstract

Although studies have demonstrated that Large Language Models (LLMs) can perform well on Out-of-Distribution (OOD) tasks, their advantage tends to diminish as the distribution shift becomes more severe. Consequently, researchers aim to retrieve distributionally similar and informative demonstrations from the available source domain to boost the inference capabilities of LLMs. However, in practical scenarios where the target domain is inaccessible, evaluating the unknown distribution is challenging, which indirectly impacts the quality of the selected demonstrations. To address this problem, we propose \textbf{DOPA}, a demonstration search framework that incorporates an OOD proxy to approximate the inaccessible target domain and guide the retrieval process. Building on proxy-based evaluation, DOPA further introduces a Mahalanobis distance-based global diversity constraint to ensure sufficient diversity among the retrieved demonstrations. Experimental results on multiple LLMs and tasks demonstrate that DOPA effectively enhances robustness in OOD settings\footnote{https://github.com/bort64/ood\_code}.

Figures

Figures reproduced from arXiv: 2606.00014 by the authors.

**Figure 2.** Performance influence of k on LLaMA3.2-3B and Qwen3-1.7B across tasks. 1 2 3 4 5 Samples 55 60 65 Accuracy (%) SA LLaMA3.2-3B (DOPA) LLaMA3.2-3B (KNN) Qwen3-1.7B (DOPA) Qwen3-1.7B (KNN) 1 2 3 4 5 Samples 52 54 56 58 60 TD 1 2 3 4 5 Samples 37 38 39 40 NLI 1 2 3 4 5 Samples 30 35 40 45 NER [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Performance influence of N on DOPA and KNN based on LLaMA3.2-3B and Qwen3-1.7B, the shaded areas with corresponding colors indicate the performance differences. and hinder generalization, whereas larger k values introduce noisy or redundant demonstrations that may degrade performance. Based on systematic evaluations across multiple tasks and datasets, we adopt k = 800 as a unified setting, as it provides a favorable… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Different visualization results on sst. based energy scores (Liu et al., 2020) and estimating their distributions via kernel density estimation (KDE) (Wkeglarczyk, 2018). We adopt energyscore distributions rather than the commonly used t-SNE visualizations, as repres…

**Figure 5.** Figure 5: Different KDE visualization results on all classification tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 6.** Figure 6: Case study on sst and implicit. strates slightly better diversity: in sst, while both methods retrieve strongly negative, concise opinions, DOPA’s samples vary slightly more in content and phrasing. In the implicit task, both methods capture politically charged and pr…

**Figure 7.** Figure 7: Euclidean distance comparison to target domain samples for retrieval results with and without the diversity [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[1]

A survey on evaluation of large language mod- els.ACM Trans. Intell. Syst. Technol., 15(3):39:1– 39:45. Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. UPRISE: universal prompt retrieval for improving zero-shot evaluation. InProceedings of the 2023 Conference on Empirical...

arXiv 2023
[2]

Location

Revisiting demonstration selection strategies in in-context learning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9090– 9101. Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V . Dillon, and ...

arXiv 2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

A survey on evaluation of large language mod- els.ACM Trans. Intell. Syst. Technol., 15(3):39:1– 39:45. Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. UPRISE: universal prompt retrieval for improving zero-shot evaluation. InProceedings of the 2023 Conference on Empirical...

arXiv 2023

[2] [2]

Location

Revisiting demonstration selection strategies in in-context learning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9090– 9101. Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V . Dillon, and ...

arXiv 2024