Pith. sign in

REVIEW 4 major objections 4 minor 61 references

Exploring Explanations Improves the Robustness of In-Context Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generating and comparing reasons for every possible label makes in-context learning more robust to out-of-distribution data, beating plain ICL and single-explanation X-ICL on most OOD benchmarks across five LLMs.

desk verdict A useful, well-ablated extension of X-ICL whose headline claim outruns the statistics, and whose mechanism is plausible but not pinned down. read the letter →

arxiv 2506.02378 v1 pith:2557SX67 submitted 2025-06-03 cs.CL cs.AIcs.LG

Ukyo Honda , Tatsushi Oka This is my paper
classification cs.CLcs.AIcs.LG
keywords in-contextlearningout-of-distributionrobustnessexplanationsnaturallanguageinferenceparaphraseidentificationlatentvariablemodelchain-of-thoughtself-consistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces X2-ICL, a prompting strategy that asks a language model to write a separate explanation for every candidate label of a test input, then select the label whose explanation seems most valid. This is contrasted with standard in-context learning (ICL), which predicts directly from demonstrations, and with the earlier X-ICL, which generates a single explanation for the observed label. The authors claim that systematically exploring all label-conditioned reasoning paths substantially improves robustness when the test distribution is adversarially shifted from the demonstration distribution, while acknowledging reduced performance on in-distribution data and higher token costs. If the claim holds, it means that prompting models to enumerate and compare alternative rationales is a cheap, fine-tuning-free way to harden few-shot classification against distribution shift.

What carries the argument

The central object is the label-conditioned reasoning path: for every demonstration and test input, the model is prompted to write a block of 'Possible Reasoning for {label}' for each label in the finite set $\mathcal{Y}$, followed by the instruction to explore all labels and select the one with the most valid reasoning. This implements a latent-variable model in which each label $\ell$ has its own latent explanation $r_\ell$, and the comparison step decides the final prediction. The mechanism forces the model to articulate and weigh alternative accounts of the input, rather than committing to the first reasoning path that matches the observed demonstration pattern.

What would settle it

A reader could test whether the comparison step, rather than the extra text, causes the gains by taking the X2-ICL prompt and randomizing or deleting the 'Possible Reasoning' blocks at test time while keeping all other content identical; if OOD accuracy stays roughly the same, the claimed validity-comparison mechanism is not what drives the improvement.

Watch

Extended reading notes

Core claim

The central claim is that covering the reasoning space for all labels, not just the label that appears in the demonstrations, yields more reliable out-of-distribution predictions. The paper formalizes each explanation as a latent variable $r_\ell$ drawn conditionally on the label $\ell$, and models the joint distribution as $p(y, r|x) = p(y|r, x)p(r|x)$, with the decision rule selecting the label whose drawn reasoning is most valid. Empirically, X2-ICL outperforms both ICL and X-ICL on six to eight of eight OOD datasets for each of five different LLMs, with stronger gains on higher-performing models, and it improves on the stochastic multi-path baseline (self-consistency) at equal path counts. The paper also shows that a simple instruction to explore reasons without demonstration does not replicate the effect, indicating that the explanation-augmented demonstrations are essential to guide the structured reasoning.

Load-bearing premise

The method assumes that the explanations a model generates for labels it never saw in the demonstrations are informative, and that the final step actually compares their validity rather than picking a confident-sounding story; Section 4.5 documents exactly such failure cases.

Editorial extensions

If this is right

  • X2-ICL gives consistent OOD accuracy gains over both ICL and X-ICL across closed- and open-source models, suggesting the benefit is not tied to one model family.
  • Systematic, label-structured exploration beats stochastic self-consistency when both use the same number of reasoning paths, implying that the organization of the search matters more than raw sampling.
  • The comparison step, not merely the presence of extra text, drives the gains: a zero-shot chain-of-thought instruction that asks for multi-label exploration performs no better than plain zero-shot CoT.
  • The approach trades in-distribution accuracy and token cost for OOD robustness, a trade-off the paper explicitly flags as an open problem.
  • Diversity in the reasoning space yields larger OOD gains than diversity in the demonstration space, based on the comparison with retrieval-based example selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper limits X2-ICL to a fixed label space, a natural extension would be to adapt the same generate-and-compare loop to open-ended generation by having the model propose candidate answers, then generate and compare rationales for each candidate.
  • The documented failure modes (encyclopedic overreach, over-inference) suggest that adding a validity check—for example, requiring the winning explanation to cite only information present in the input—could further improve robustness.
  • The observed ID/OOD trade-off hints at an adaptive gating rule: apply X2-ICL when the model's confidence under ordinary ICL is low, and fall back to the cheaper single-explanation path otherwise.
  • The latent-variable framing predicts that gains should grow with the number of labels up to a point, since more alternatives mean broader coverage of the reasoning space; this is a testable extension the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces X2-ICL, an in-context learning method that augments few-shot demonstrations with LLM-generated explanations for every possible label rather than only the observed label, and instructs the model at inference to generate rationales for all labels and select the label with the most valid reasoning. The authors formalize the approach in a latent-variable classification framework and evaluate it on natural language inference and paraphrase identification using five LLMs and eight out-of-distribution datasets, reporting that X2-ICL outperforms both ICL and X-ICL on six to eight of eight OOD datasets for each model. Additional experiments compare against retrieval-based ICL, self-consistency, instruction-only controls, zero-shot chain-of-thought, and non-explanation robustness methods, and the paper includes qualitative success/failure analysis and cost measurements.

Significance. If the proposed mechanism were validated, X2-ICL would be an annotation-light and training-free method for improving OOD robustness in ICL, and the breadth of evaluation across five models, eight OOD sets, and several ablations is a genuine strength. The public code, the explicit cost analysis, and the candid failure-case discussion are also valuable. However, the current evidence supports a prompt-engineering effect more strongly than the claimed latent-variable exploration mechanism: the implemented inference procedure does not match the formal label-conditioned sampling in Eq. (13), no explanation-validity metric is applied in the main evaluation, and the headline comparisons lack significance testing.

major comments (4)
  1. [§3.3, Algorithm 1, Figure 2] The inference stage as implemented does not execute the label-conditioned rationale sampling described in Eq. (13). Algorithm 1 line 14 draws a single joint latent vector r′ from p̂(r|x′), whereas Eq. (13) defines each r_ℓ as a sample from p̃(r_ℓ | y=ℓ, x). The actual prompt in Figure 2 generates “Possible Reasoning for [label]” segments for all three labels in one autoregressive pass, with no per-label conditioning step; the model is never forced to produce a rationale that actually supports the label it is attached to. Consequently, the comparison step in lines 15–17 selects the label whose generated text “seems most valid” per the instruction rather than the label whose rationale is a faithful sample from the label-conditioned distribution. Section 4.5 provides direct evidence of unfaithful rationales: in Figures 10 and 12 the text labeled “contradiction” argues against the contradiction label, and in Figure 12 the “contradiction” rationale states that the hypothesis cannot be concluded. The paper should either modify the method to perform per-label sampling, or revise the formal model to describe the actual single-pass generation, and should show that the generated rationales support their assigned labels before claiming that X2-ICL explores label-conditioned explanations.
  2. [§4.4, §4.5] The main evaluation contains no measure of explanation validity; the only quality signal is end-task accuracy. The proposed mechanism assumes that the rationales are meaningful latent variables that justify their labels, but Section 4.5 documents several cases where the winning rationale is plausible yet unsupported or even self-contradictory. Without a validity metric, such as human annotation of a sample of rationales or an automatic label-consistency check, applied in the main evaluation, the observed OOD gains could be driven by the selection instruction “select the label that has the most valid reasoning” or simply by the additional text generated, rather than by the exploration of label-conditioned explanations. The ablation in Section 4.4(ii) controls for the label-space information in the instruction by adding an instruction to X-ICL, but it does not remove the selection instruction from X2-ICL or replace it with a neutral instruction, so it does not isolate the effect of the instruction from the effect of generating multiple rationales. An additional control, for example prompting the model to generate the same amount of text without per-label rationales and then select the most valid label, or ablating the “most valid reasoning” phrase, would strengthen the causal claim.
  3. [Table 1, §4.2] The central claim that X2-ICL outperforms both ICL and X-ICL on six to eight out of eight OOD datasets for each of five different LLMs rests on mean accuracy differences computed over only four random demonstration seeds. No significance tests, paired comparisons, or confidence intervals are reported, and many of the differences are within one standard deviation. For example, Phi-4-14B on HANS improves from 84.05±1.90 for X-ICL to 84.45±1.06 for X2-ICL, GPT-4o on NAN improves from 78.29±1.23 to 78.78±0.37, and DeepSeek-R1-8B on PISP improves from 67.25±1.35 to 68.73±1.96. With only four seeds, these differences are not convincing evidence of systematic improvement. The paper should report paired significance tests across the four seeds, or use more seeds, and should state which of the eight OOD datasets show statistically significant gains for each model; the “six to eight” summary should be accompanied by the number of significant wins.
  4. [§4.5, Table 1] The qualitative analysis suggests that X2-ICL’s improvement may be partly a label-shift artifact rather than a general robustness gain. In the 42 ANLI R1 instances where X-ICL and X2-ICL disagree, 24 move from neutral to entailment/contradiction while only 14 move in the opposite direction, and the in-distribution SNLI/QQP results show large drops, for example Gemini-1.5-Pro on SNLI falls from 89.80 for ICL to 82.70 for X2-ICL, and DeepSeek-R1-8B on QQP falls from 48.55 to 41.95. If X2-ICL systematically reduces the frequency of neutral predictions, it will appear to improve on OOD datasets whose gold labels are mostly non-neutral and to hurt on datasets with many neutral labels. The paper should report per-label accuracy or at least the label distribution of errors for each dataset, and should check whether the OOD gains survive when the label distribution is controlled or when evaluation is restricted to examples whose gold label is neutral.
minor comments (4)
  1. [§2.1, Algorithm 1, Algorithm 3] There are several typos and minor notation issues: “minimizing the the expected loss” in Section 2.1, “Augumented data” in Algorithm 1 line 9 and Algorithm 3 line 6, and the factorization in Eq. (14)/Algorithm 1 line 11 is introduced but never used in the inference procedure; please revise for consistency.
  2. [§4.2] The sentence “X2-ICL outperforms both ICL and X-ICL on six to eight out of eight OOD datasets for each of five different LLMs” is imprecise, because for DeepSeek-R1-8B, X2-ICL is below X-ICL on ANLI R3 and PAWS; please specify the per-dataset comparisons or qualify the claim.
  3. [§4.4(i), Appendix C] The self-consistency baseline is run with sampling temperature 0.7 while X2-ICL is run with temperature 0 for GPT-4o; please clarify whether this decoding-temperature difference could confound the comparison between structured exploration and stochastic sampling.
  4. [Figures 10–12] The figure captions could state more explicitly that the “contradiction” rationale in each example argues against the contradiction label, since the body text already makes this point but the captions are ambiguous on their own.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the formal derivation is descriptive, Eq. (15) is the algorithm's definition rather than a fitted prediction, and OOD robustness is tested against external benchmarks.

full rationale

The derivation chain is descriptive rather than circular. In Section 2.2, Eq. (5), p(y|x) = integral p(y|r,x)p(r|x) dr, is the law of total probability, and Eq. (14) is the definition of conditional probability; both serve only to introduce notation for the latent-variable framing. The proposed decision rule, delta_X2-ICL(x') := arg max p_hat(y' | r', x') (Eq. 15; Algorithm 1, lines 14-17), is introduced with ':=' as the algorithm's own definition, not derived as a theorem whose conclusion equals its premise. No parameter is fitted to the OOD evaluation sets, so the central empirical claim in Section 4.2 is not a fitted input renamed as a prediction: X2-ICL could have underperformed and indeed does underperform X-ICL on several in-distribution datasets (SNLI, QQP) and on some OOD sets (e.g., Gemini-2.0-Flash on ANLI R1). The ablations in Section 4.4 (i)-(iii) isolate the mechanism against self-consistency, instruction-only, and zero-shot CoT baselines, and the meta-prompts are borrowed transparently from external work (He et al., 2024). Two mildly self-referential elements exist but do not carry the argument: (a) one self-citation in Related Work, 'Analogous to X2-ICL, Honda et al. (2024) modeled various latent features and utilized them for prediction,' which is illustrative and non-load-bearing; and (b) GPT-4o generates the explanations incorporated into demonstrations and is also one of the evaluated models (Section 4.1), a potential same-model-family confound rather than a definitional identity. The documented failure modes in Section 4.5 (encyclopedic overreach, unfaithful rationale selection) bear on rationale validity and correctness risk, not on circularity. Overall, the paper is self-contained against external benchmarks and exhibits no step where an output equals its input by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on no fitted constants; the main debts are domain assumptions about LLM explanations behaving as latent variables and about benchmark labels capturing distribution shift. The instruction and 8-shot setting are hand-chosen hyperparameters. No new physical or mechanistic entities are introduced.

free parameters (3)
  • n_shots = 8
    Fixed following X-ICL protocol; results are specific to the 8-shot setting, and no sensitivity analysis is provided.
  • decoding_temperature = 0 (closed APIs best effort; greedy for open models)
    Chosen for reproducibility; closed API 'deterministic' decoding is not guaranteed, so exact outputs may vary across service versions.
  • x2_icl_instruction = Explore the reasoning behind all the labels. Then, select the label that has the most valid reasoning.
    The instruction is a hand-designed component of the method. The ablation in Section 4.4(ii) tests a label-space-only variant but not alternative phrasings, so the exact wording is a free design choice.
assumptions (5)
  • standard math The Bayes classifier argmax_y p(y|x) minimizes 0-1 loss as stated in Eq. (4).
    Used in Section 2.1 as the target decision rule; standard result, not proved in the paper.
  • domain assumption LLM in-context learning corresponds to estimating p_hat(y|x) = p(y|x, D_n).
    Section 3.1 treats ICL as implicit maximum-conditional-probability estimation without a proof that LLM predictions obey this probabilistic model.
  • domain assumption Explanations sampled with the meta-prompt approximate p(r|y,x).
    Section 3.2 Eqs. (9)-(10) and Section 3.3 Eq. (13); this is the bridge between generated text and latent variables and is untested beyond qualitative examples.
  • domain assumption At test time the model can generate and compare all-label reasoning and select the best-supported label.
    Algorithm 1 lines 14-17 and Eq. (15); Section 4.5 shows failures of exactly this step.
  • domain assumption Selected OOD test sets have correct labels and represent adversarial distribution shifts.
    Section 4.1 relies on manual curation and verification claims; no label audit is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Explanations Improves the Robustness of In-Context Learning." pith.science (2026). https://pith.science/paper/2557SX67

@misc{pith2026250602378,
  author       = {Pith},
  title        = {Pith review of: Exploring Explanations Improves the Robustness of In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2557SX67}},
  note         = {Machine review of arXiv:2506.02378}
}
abstract

In-context learning (ICL) has emerged as a successful paradigm for leveraging large language models (LLMs). However, it often struggles to generalize beyond the distribution of the provided demonstrations. A recent advancement in enhancing robustness is ICL with explanations (X-ICL), which improves prediction reliability by guiding LLMs to understand and articulate the reasoning behind correct labels. Building on this approach, we introduce an advanced framework that extends X-ICL by systematically exploring explanations for all possible labels (X$^2$-ICL), thereby enabling more comprehensive and robust decision-making. Experimental results on multiple natural language understanding datasets validate the effectiveness of X$^2$-ICL, demonstrating significantly improved robustness to out-of-distribution data compared to the existing ICL approaches.

Figures

Figures reproduced from arXiv: 2506.02378 by the authors.

Figure 1
Figure 1. Overview of ICL, X-ICL, and our proposed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of X2 -ICL prompt for NLI. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. An example of X2 -ICL prompt for paraphrase identification. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Meta-prompt for generating explanations to be included in NLI demonstrations. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Meta-prompt for generating explanations to be included in paraphrase-identification demonstrations. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: An example in which X2 -ICL provides the correct answer, whereas X-ICL fails. X2 -ICL enabled the model to carefully examine the details of the input and to reason based on confidently inferred information. Successful case of X2 -ICL Premise: David Carter (born Novembe…
Figure 7
Figure 7. Figure 7: An example in which X2 -ICL provides the correct answer, whereas X-ICL fails. X2 -ICL enabled the model to carefully examine the details of the input and to reason based on confidently inferred information. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: An example in which X2 -ICL provides the correct answer, whereas X-ICL fails. X2 -ICL prevented the model from making hasty inferences by drawing attention to the lack of decisive information in the given context. Successful case of X2 -ICL Premise: The Best of David B…
Figure 9
Figure 9. Figure 9: An example in which X2 -ICL provides the correct answer, whereas X-ICL fails. X2 -ICL prevented the model from making hasty inferences by drawing attention to the lack of decisive information in the given context. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: An example in which X-ICL provides the correct answer, whereas X [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: An example in which X-ICL provides the correct answer, whereas X [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: An example in which X-ICL provides the correct answer, whereas X [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: An example in which X-ICL provides the correct answer, whereas X [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. https://arxiv.org/abs/2412.08905 Phi-4 technical report . arXiv preprint arXiv:2412.08905v1

    arXiv 2024

  2. [2]

    Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. https://doi.org/10.48550/arXiv.1907.02893 Invariant risk minimization . arXiv preprint arXiv:1907.02893v3

    arXiv doi:10.48550/arxiv.1907.02893 2019

  3. [3]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1075 2015

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://proceedings.neurips.cc/paper_fil...

    2020

  5. [5]

    Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. 2022. https://doi.org/10.18653/v1/2022.naacl-main.278 Can rationalization improve robustness? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3792--3805, Seattle, United States. Associatio...

    work page doi:10.18653/v1/2022.naacl-main.278 2022

  6. [6]

    Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. https://doi.org/10.18653/v1/D19-1418 Don't take the easy way out: Ensemble based methods for avoiding known dataset biases . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-...

    doi:10.18653/v1/d19-1418 2019

  7. [7]

    Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. 2021. https://proceedings.mlr.press/v139/creager21a.html Environment inference for invariant learning . In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2189--2200. PMLR

    work page 2021

  8. [8]

    Dempster, Nan M

    Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x Maximum likelihood from incomplete data via the em algorithm . Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1--38

    arXiv 1977

Show all 61 references
  1. [9]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.64 A survey on in-context learning . In Proceedings of the 2024 Conference on E...

    2024 doi

  2. [10]

    Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/D19-1107 Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets . In Proceedings of the 2019 Conference on Empirical Methods in Nat...

    2019 doi

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv...

    2025 arXiv

  4. [12]

    Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.930 Coverage-based example selection for in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924--13950, Singapore. Associ...

    2023 doi

  5. [13]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-2017 Annotation artifacts in natural language inference data . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  6. [14]

    He He, Sheng Zha, and Haohan Wang. 2019. https://doi.org/10.18653/v1/D19-6115 Unlearn dataset bias in natural language inference by fitting the residual . In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132--142, Hong Ko...

    2019 doi

  7. [15]

    Xuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini, and Pontus Stenetorp. 2024. https://doi.org/10.18653/v1/2024.acl-long.728 Using natural language explanations to improve robustness of in-context learning . In Proceedings of the 62nd Annual Meeting of the Associat...

    2024 doi

  8. [16]

    Ukyo Honda, Tatsushi Oka, Peinan Zhang, and Masato Mita. 2024. https://doi.org/10.1162/tacl_a_00701 Not eliminate but aggregate: Post-hoc control over mixture-of-experts to address shortcut shifts in natural language understanding . Transactions of the Association for Computat...

    2024 doi

  9. [17]

    Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs First quora dataset release: Question pairs . Accessed on Feb. 9, 2025

    2017

  10. [18]

    Joonwon Jang, Sanghwan Jang, Wonbin Kweon, Minjin Jeon, and Hwanjo Yu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.242 Rectifying demonstration shortcut in in-context learning . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Co...

    2024 doi

  11. [19]

    Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. https://doi.org/10.1162/tacl_a_00523 Investigating reasons for disagreement in natural language inference . Transactions of the Association for Computational Linguistics, 10:1357--1374

    2022 doi

  12. [20]

    Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. https://doi.org/10.18653/v1/2020.acl-main.769 End-to-end bias mitigation by modelling biases in corpora . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706...

    2020 doi

  13. [21]

    Pride Kavumba, Ana Brassard, Benjamin Heinzerling, and Kentaro Inui. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.162 Prompting for explanations improves adversarial NLI . is this true? \ yes \ it is \ true \ because \ it weakens superficial cues \ . In Findings of the...

    2023 doi

  14. [22]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...

    2022

  15. [23]

    Itay Levy, Ben Bogin, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.78 Diverse demonstrations improve in-context compositional generalization . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023 doi

  16. [24]

    Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, and Tat-Seng Chua. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.95 Robust prompt optimization for large language models against distribution shifts . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  17. [25]

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. 2021. https://proceedings.mlr.press/v139/liu21f.html Just train twice: Improving group robustness without training group information . In Proceedings of the...

    2021

  18. [26]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  19. [27]

    Tianyu Liu, Zheng Xin, Baobao Chang, and Zhifang Sui. 2020 a . https://aclanthology.org/2020.lrec-1.846/ H ypo NLI : Exploring the artificial patterns of hypothesis-only bias in natural language inference . In Proceedings of the Twelfth Language Resources and Evaluation Confer...

    2020

  20. [28]

    Tianyu Liu, Zheng Xin, Xiaoan Ding, Baobao Chang, and Zhifang Sui. 2020 b . https://doi.org/10.18653/v1/2020.conll-1.48 An empirical study on model-agnostic debiasing strategies for robust natural language inference . In Proceedings of the 24th Conference on Computational Natu...

    2020 doi

  21. [29]

    Duncan Luce

    R. Duncan Luce. 1959. Individual Choice Behavior: A Theoretical Analysis. Wiley, New York. Reprinted by Dover Publications, 2005

    1959

  22. [30]

    Josh Magnus Ludan, Yixuan Meng, Tai Nguyen, Saurabh Shah, Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2023. https://doi.org/10.18653/v1/2023.acl-long.242 Explanation-based finetuning makes models more robust to spurious cues . In Proceedings of the 61st Annual Mee...

    2023 doi

  23. [31]

    Jacob Marschak. 1960. Binary choice constraints and random utility indicators. Stanford Research Memorandum

    1960

  24. [32]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2019 doi

  25. [33]

    Daniel McFadden. 1973. Conditional logit analysis of qualitative choice behavior. In P. Zarembka, editor, Frontiers in Econometrics, pages 105--142. Academic Press

    1973

  26. [34]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  27. [35]

    Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024. https://doi.org/10.18653/v1/2024.naacl-long.267 In-context learning generalizes, but not always robustly: The case of syntax . In Proceedings of the 2024 Conference of the North American Chapter of the Associat...

    2024 doi

  28. [36]

    Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. https://aclanthology.org/C18-1198/ Stress test evaluation for natural language inference . In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340--...

    2018

  29. [37]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for C...

    2020 doi

  30. [38]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020 b . https://doi.org/10.18653/v1/2020.emnlp-main.734 What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMN...

    2020 doi

  31. [39]

    Ellie Pavlick and Tom Kwiatkowski. 2019. https://doi.org/10.1162/tacl_a_00293 Inherent disagreements in human textual inferences . Transactions of the Association for Computational Linguistics, 7:677--694

    2019 doi

  32. [40]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  33. [41]

    Hashimoto, and Percy Liang

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. https://openreview.net/forum?id=ryxGuJrFvS Distributionally robust neural networks . In International Conference on Learning Representations

    2020

  34. [42]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. https://proceedings.mlr.press/v202/santurkar23a.html Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, volume...

    2023

  35. [43]

    Charlotte Siska, Katerina Marazopoulou, Melissa Ailem, and James Bono. 2024. https://doi.org/10.18653/v1/2024.acl-long.560 Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks . In Proceedings of the 62nd Annual Meeting of the Association ...

    2024 doi

  36. [44]

    Joe Stacey, Yonatan Belinkov, and Marek Rei. 2022. https://doi.org/10.1609/aaai.v36i10.21386 Supervising model attention with human explanations for robust natural language inference . Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11349--11357

    2022 doi

  37. [45]

    Zechen Sun, Yisheng Xiao, Juntao Li, Yixin Ji, Wenliang Chen, and Min Zhang. 2024. https://aclanthology.org/2024.lrec-main.602/ Exploring and mitigating shortcut learning for generative large language models . In Proceedings of the 2024 Joint International Conference on Comput...

    2024

  38. [46]

    Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. https://doi.org/10.18653/v1/2023.findings-acl.284 Large language models can be lazy learners: Analyze shortcuts in in-context learning . In Findings of the Association for Computational Linguistics: ACL 2023, pages 4...

    2023 doi

  39. [47]

    L. L. Thurstone. 1927. https://doi.org/10.1037/h0070288 A law of comparative judgment . Psychological Review, 34(4):273--286

    1927 doi

  40. [48]

    Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022. https://doi.org/10.18653/v1/2022.aacl-main.65 Not another negation benchmark: The N a N - NLI test suite for sub-clausal negation . In Proceedings of the 2nd Conference of ...

    2022 doi

  41. [49]

    Jindong Wang, Xixu HU, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. 2023 a . https://openreview.net/forum?id=uw6HSkgoM29 On the robustness of chat GPT : An adversarial and out-of-distribut...

    2023

  42. [50]

    Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. 2023 b . https://arxiv.org/abs/2305.14950 Adversarial demonstration attacks on large language models . arXiv preprint arXiv:2305.14950v2

    2023 arXiv

  43. [51]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 c . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confer...

    2023

  44. [52]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  45. [53]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  46. [54]

    Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. 2023. https://proceedings.mlr.press/v202/yang23s.html Change is hard: A closer look at subpopulation shift . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machi...

    2023

  47. [55]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. https://proceedings.mlr.press/v202/ye23c.html Compositional exemplars for in-context learning . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machin...

    2023

  48. [56]

    Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.679 Do LLM s overcome shortcut learning? an evaluation of shortcut challenges in large language models . In Proceedings of the 2024 Conference on Empirical Methods in ...

    2024 doi

  49. [57]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  50. [58]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. https://doi.org/10.18653/v1/N19-1131 PAWS : Paraphrase adversaries from word scrambling . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Tec...

    2019 doi

  51. [59]

    Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.28 Explore spurious correlations at the concept level in language models for text classification . In Proceedings of the 62nd Annual Meeting of the Associati...

    2024 doi

  52. [60]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.