{"id":"5bf28f93-9a62-4df9-94c1-5853c29731a4","arxiv_id":"2506.02378","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Generating and comparing label-by-label explanations in the prompt improves out-of-distribution accuracy of in-context learning on most tested NLU benchmarks, at the cost of in-distribution performance and higher inference cost.","lead":"X2-ICL is a prompting trick that makes an AI think through a reason for every possible answer before picking one. In tests across five large language models and eight shifted benchmarks, it usually beats simpler in-context learning, though it can hurt ordinary in-distribution accuracy and costs more tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"X2-ICL's inference does not implement the label-conditioned rationale sampling of Eq. 13, and no validity check shows the rationales support their labels, so the claimed 'exploration of explanations' mechanism is unverified.","rationale":"I focused on the mechanism rather than the statistics because the empirical pattern is consistent across many dataset-model pairs, whereas the paper's theoretical contribution—that label-conditioned reasoning explains the gains—is directly contradicted by the implementation gap between Eq. 13 and Algorithm 1. The failure cases in Section 4.5 supply concrete evidence that rationales are often unfaithful to their assigned labels, so the 'exploration of explanations' is not verified. This aligns with the reader's weakest_assumption, and the reader's conditional verdict already captures the need for additional validation. The proposed concrete test would isolate whether the single-pass generation used in the main experiments actually implements label-conditioned exploration, and whether the rationales carry the information the method claims. If the independent-conditioning control matches or beats the current implementation, the method's mechanism is credible; if not, the paper should be revised to describe what is actually happening. I do not see a reason to move the verdict to REJECT or ACCEPT; CONDITIONAL remains appropriate, and since the reader already recommended CONDITIONAL, I mark the verdict as unchanged.","tokens_in":23695,"tokens_out":11355,"duration_ms":104646,"concrete_test":"Run a controlled experiment on ANLI-R1/R2/R3 with GPT-4o over the same four seeds: (a) current single-pass X2-ICL as in Figure 2; (b) independent per-label rationale generation, where for each test example the model is separately prompted to produce a reason why label ℓ is correct (conditioned on ℓ as in Eq. 13), followed by a separate selection step; (c) current X2-ICL with the instruction removed; and (d) X-ICL with the X2-ICL instruction added. If (b) does not match or beat (a), the label-conditioned exploration is not faithfully implemented, and the gains may be attributable to the instruction or extra decoding rather than to label-conditioned reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that generating and comparing label-conditioned rationales for all labels is what drives OOD robustness. Two linked gaps threaten this. First, the inference procedure does not implement the formal model. Algorithm 1 line 14 draws r' ~ p̂(r|x') as a single joint latent vector, and the actual prompt (Figure 2) elicits 'Possible Reasoning for [label]' in one autoregressive pass with no per-label conditioning. In contrast, the demonstrations are built by sampling each r_{i,ℓ} from p̃(r_ℓ | y=ℓ, x_i) (Eq. 13). At test time nothing conditions each generated rationale on its label, so the r'_ℓ cannot be assumed to be samples from p(r_ℓ | y=ℓ, x'). The comparison step then selects the label whose rationale 'seems most valid', a linguistic judgment over rationales that may not be label-specific. Section 4.5 provides direct evidence: in Figures 10 and 12 the rationale labeled 'contradiction' argues against contradiction, and Figure 12's 'contradiction' rationale states the hypothesis cannot be concluded. Thus the method as implemented is not necessarily 'exploring explanations for all possible labels'; it may simply be generating extra text plus an instruction. Second, no validity metric is ever applied to the rationales in the main evaluation; the only quality signal is end-task accuracy. Given the documented unfaithful rationales, the improvement could stem from the instruction ('select the label that has the most valid reasoning') or from additional decoding, rather than from the label-conditioned explanations that the paper's theory and title emphasize. These concerns are load-bearing because if the rationales are not faithful, the method's proposed mechanism—and the reason to expect it to generalize to new OOD datasets—is unsupported, even if the mean accuracies in Table 1 are reproduced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces X2-ICL, an in-context learning method that augments few-shot demonstrations with LLM-generated explanations for every possible label rather than only the observed label, and instructs the model at inference to generate rationales for all labels and select the label with the most valid reasoning. The authors formalize the approach in a latent-variable classification framework and evaluate it on natural language inference and paraphrase identification using five LLMs and eight out-of-distribution datasets, reporting that X2-ICL outperforms both ICL and X-ICL on six to eight of eight OOD datasets for each model. Additional experiments compare against retrieval-based ICL, self-consistency, instruction-only controls, zero-shot chain-of-thought, and non-explanation robustness methods, and the paper includes qualitative success/failure analysis and cost measurements.","tokens_in":24047,"tokens_out":7277,"duration_ms":58153,"significance":"If the proposed mechanism were validated, X2-ICL would be an annotation-light and training-free method for improving OOD robustness in ICL, and the breadth of evaluation across five models, eight OOD sets, and several ablations is a genuine strength. The public code, the explicit cost analysis, and the candid failure-case discussion are also valuable. However, the current evidence supports a prompt-engineering effect more strongly than the claimed latent-variable exploration mechanism: the implemented inference procedure does not match the formal label-conditioned sampling in Eq. (13), no explanation-validity metric is applied in the main evaluation, and the headline comparisons lack significance testing.","major_comments":[{"comment":"The inference stage as implemented does not execute the label-conditioned rationale sampling described in Eq. (13). Algorithm 1 line 14 draws a single joint latent vector r′ from p̂(r|x′), whereas Eq. (13) defines each r_ℓ as a sample from p̃(r_ℓ | y=ℓ, x). The actual prompt in Figure 2 generates “Possible Reasoning for [label]” segments for all three labels in one autoregressive pass, with no per-label conditioning step; the model is never forced to produce a rationale that actually supports the label it is attached to. Consequently, the comparison step in lines 15–17 selects the label whose generated text “seems most valid” per the instruction rather than the label whose rationale is a faithful sample from the label-conditioned distribution. Section 4.5 provides direct evidence of unfaithful rationales: in Figures 10 and 12 the text labeled “contradiction” argues against the contradiction label, and in Figure 12 the “contradiction” rationale states that the hypothesis cannot be concluded. The paper should either modify the method to perform per-label sampling, or revise the formal model to describe the actual single-pass generation, and should show that the generated rationales support their assigned labels before claiming that X2-ICL explores label-conditioned explanations.","section":"§3.3, Algorithm 1, Figure 2"},{"comment":"The main evaluation contains no measure of explanation validity; the only quality signal is end-task accuracy. The proposed mechanism assumes that the rationales are meaningful latent variables that justify their labels, but Section 4.5 documents several cases where the winning rationale is plausible yet unsupported or even self-contradictory. Without a validity metric, such as human annotation of a sample of rationales or an automatic label-consistency check, applied in the main evaluation, the observed OOD gains could be driven by the selection instruction “select the label that has the most valid reasoning” or simply by the additional text generated, rather than by the exploration of label-conditioned explanations. The ablation in Section 4.4(ii) controls for the label-space information in the instruction by adding an instruction to X-ICL, but it does not remove the selection instruction from X2-ICL or replace it with a neutral instruction, so it does not isolate the effect of the instruction from the effect of generating multiple rationales. An additional control, for example prompting the model to generate the same amount of text without per-label rationales and then select the most valid label, or ablating the “most valid reasoning” phrase, would strengthen the causal claim.","section":"§4.4, §4.5"},{"comment":"The central claim that X2-ICL outperforms both ICL and X-ICL on six to eight out of eight OOD datasets for each of five different LLMs rests on mean accuracy differences computed over only four random demonstration seeds. No significance tests, paired comparisons, or confidence intervals are reported, and many of the differences are within one standard deviation. For example, Phi-4-14B on HANS improves from 84.05±1.90 for X-ICL to 84.45±1.06 for X2-ICL, GPT-4o on NAN improves from 78.29±1.23 to 78.78±0.37, and DeepSeek-R1-8B on PISP improves from 67.25±1.35 to 68.73±1.96. With only four seeds, these differences are not convincing evidence of systematic improvement. The paper should report paired significance tests across the four seeds, or use more seeds, and should state which of the eight OOD datasets show statistically significant gains for each model; the “six to eight” summary should be accompanied by the number of significant wins.","section":"Table 1, §4.2"},{"comment":"The qualitative analysis suggests that X2-ICL’s improvement may be partly a label-shift artifact rather than a general robustness gain. In the 42 ANLI R1 instances where X-ICL and X2-ICL disagree, 24 move from neutral to entailment/contradiction while only 14 move in the opposite direction, and the in-distribution SNLI/QQP results show large drops, for example Gemini-1.5-Pro on SNLI falls from 89.80 for ICL to 82.70 for X2-ICL, and DeepSeek-R1-8B on QQP falls from 48.55 to 41.95. If X2-ICL systematically reduces the frequency of neutral predictions, it will appear to improve on OOD datasets whose gold labels are mostly non-neutral and to hurt on datasets with many neutral labels. The paper should report per-label accuracy or at least the label distribution of errors for each dataset, and should check whether the OOD gains survive when the label distribution is controlled or when evaluation is restricted to examples whose gold label is neutral.","section":"§4.5, Table 1"}],"minor_comments":[{"comment":"There are several typos and minor notation issues: “minimizing the the expected loss” in Section 2.1, “Augumented data” in Algorithm 1 line 9 and Algorithm 3 line 6, and the factorization in Eq. (14)/Algorithm 1 line 11 is introduced but never used in the inference procedure; please revise for consistency.","section":"§2.1, Algorithm 1, Algorithm 3"},{"comment":"The sentence “X2-ICL outperforms both ICL and X-ICL on six to eight out of eight OOD datasets for each of five different LLMs” is imprecise, because for DeepSeek-R1-8B, X2-ICL is below X-ICL on ANLI R3 and PAWS; please specify the per-dataset comparisons or qualify the claim.","section":"§4.2"},{"comment":"The self-consistency baseline is run with sampling temperature 0.7 while X2-ICL is run with temperature 0 for GPT-4o; please clarify whether this decoding-temperature difference could confound the comparison between structured exploration and stochastic sampling.","section":"§4.4(i), Appendix C"},{"comment":"The figure captions could state more explicitly that the “contradiction” rationale in each example argues against the contradiction label, since the body text already makes this point but the captions are ambiguous on their own.","section":"Figures 10–12"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution, but the mismatch between the formal latent-variable model and the implemented prompt is the central issue. The paper’s own qualitative examples undercut the label-conditioned explanation story, so the authors should be pushed to either implement per-label generation or reframe the contribution as a prompt-engineering result. I would not recommend rejection because the method is simple, the evaluation is broad, and the issues are addressable with additional experiments and revised claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper extends X-ICL by generating and comparing label-conditioned rationales for all possible labels, and the empirical story is mostly credible—but the “significantly improved” wording outruns the statistics, and the claimed mechanism is not as cleanly verified as the formal section suggests.\n\nWhat’s new: the all-label prompting scheme plus the explicit instruction (“Explore the reasoning behind all the labels. Then, select the label that has the most valid reasoning”) is not present in the cited X-ICL work. The latent-variable framing is a re-description of the algorithm rather than a derivation with predictive content; the contribution is empirical, and that’s fine.\n\nWhat it does well: the evidence is broad—five models, eight OOD datasets, and the ablations are genuinely careful. They rule out self-consistency at matched sampling count, the instruction alone, and demonstration-free “explore CoT.” The comparison against retrieval-based ICL is fair, and the qualitative section is candid, including failure cases. Code is public.\n\nSoft spots, in order. First, no significance tests. With four seeds and 500 test examples, many individual gains are within one standard deviation. The pattern across datasets and models is consistent, but the abstract’s “significantly” should be backed by paired tests or at least a sign test. Second, the in-distribution regression is real and sometimes large—Phi-4 loses about five points on SNLI and ten on QQP. The limitations section acknowledges it but doesn’t characterize the trade-off. Third, the mechanism. The stress-test’s “no per-label conditioning” is too strong: the prompt literally precedes each rationale with “Possible Reasoning for entailment,” etc. But the rationales are all generated in a single autoregressive pass, not drawn independently per label as Eq. 13 suggests, and the paper’s own examples show rationales that argue against their stated label. No validity metric is applied. So the full package works; we don’t know the gain comes specifically from faithful label-conditioned exploration. A separate-condition or filtering experiment would pin this down. Fourth, cost is about 2.5x X-ICL; that’s acknowledged and probably acceptable for many uses.\n\nOverall this is a solid incremental contribution, not a field-changer. It deserves a serious referee who can push on significance testing and mechanism disentangling. Practitioners with small labeled sets and fixed label spaces get a free robustness boost; ICL-robustness researchers will want it as a baseline. Send it to review.","headline":"A useful, well-ablated extension of X-ICL whose headline claim outruns the statistics, and whose mechanism is plausible but not pinned down.","tokens_in":24602,"tokens_out":6948,"would_cite":false,"duration_ms":58793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generating and comparing reasons for every possible label makes in-context learning more robust to out-of-distribution data, beating plain ICL and single-explanation X-ICL on most OOD benchmarks across five LLMs.","keywords":["in-context learning","out-of-distribution robustness","explanations","natural language inference","paraphrase identification","latent variable model","chain-of-thought","self-consistency"],"falsifier":"A reader could test whether the comparison step, rather than the extra text, causes the gains by taking the X2-ICL prompt and randomizing or deleting the 'Possible Reasoning' blocks at test time while keeping all other content identical; if OOD accuracy stays roughly the same, the claimed validity-comparison mechanism is not what drives the improvement.","tokens_in":23517,"feed_emoji":"🧠","tokens_out":3945,"duration_ms":36352,"temperature":0.7,"pith_summary":"The paper introduces X2-ICL, a prompting strategy that asks a language model to write a separate explanation for every candidate label of a test input, then select the label whose explanation seems most valid. This is contrasted with standard in-context learning (ICL), which predicts directly from demonstrations, and with the earlier X-ICL, which generates a single explanation for the observed label. The authors claim that systematically exploring all label-conditioned reasoning paths substantially improves robustness when the test distribution is adversarially shifted from the demonstration distribution, while acknowledging reduced performance on in-distribution data and higher token costs. If the claim holds, it means that prompting models to enumerate and compare alternative rationales is a cheap, fine-tuning-free way to harden few-shot classification against distribution shift.","feed_headline":"Explaining every label option beats standard in-context learning","feed_subtitle":"Across five LLMs and eight OOD benchmarks, asking for each label's reasoning outperforms baseline prompting on most.","key_machinery":"The central object is the label-conditioned reasoning path: for every demonstration and test input, the model is prompted to write a block of 'Possible Reasoning for {label}' for each label in the finite set $\\mathcal{Y}$, followed by the instruction to explore all labels and select the one with the most valid reasoning. This implements a latent-variable model in which each label $\\ell$ has its own latent explanation $r_\\ell$, and the comparison step decides the final prediction. The mechanism forces the model to articulate and weigh alternative accounts of the input, rather than committing to the first reasoning path that matches the observed demonstration pattern.","core_discovery":"The central claim is that covering the reasoning space for all labels, not just the label that appears in the demonstrations, yields more reliable out-of-distribution predictions. The paper formalizes each explanation as a latent variable $r_\\ell$ drawn conditionally on the label $\\ell$, and models the joint distribution as $p(y, r|x) = p(y|r, x)p(r|x)$, with the decision rule selecting the label whose drawn reasoning is most valid. Empirically, X2-ICL outperforms both ICL and X-ICL on six to eight of eight OOD datasets for each of five different LLMs, with stronger gains on higher-performing models, and it improves on the stochastic multi-path baseline (self-consistency) at equal path counts. The paper also shows that a simple instruction to explore reasons without demonstration does not replicate the effect, indicating that the explanation-augmented demonstrations are essential to guide the structured reasoning.","pith_inferences":["Because the paper limits X2-ICL to a fixed label space, a natural extension would be to adapt the same generate-and-compare loop to open-ended generation by having the model propose candidate answers, then generate and compare rationales for each candidate.","The documented failure modes (encyclopedic overreach, over-inference) suggest that adding a validity check—for example, requiring the winning explanation to cite only information present in the input—could further improve robustness.","The observed ID/OOD trade-off hints at an adaptive gating rule: apply X2-ICL when the model's confidence under ordinary ICL is low, and fall back to the cheaper single-explanation path otherwise.","The latent-variable framing predicts that gains should grow with the number of labels up to a point, since more alternatives mean broader coverage of the reasoning space; this is a testable extension the paper does not run."],"forward_implications":["X2-ICL gives consistent OOD accuracy gains over both ICL and X-ICL across closed- and open-source models, suggesting the benefit is not tied to one model family.","Systematic, label-structured exploration beats stochastic self-consistency when both use the same number of reasoning paths, implying that the organization of the search matters more than raw sampling.","The comparison step, not merely the presence of extra text, drives the gains: a zero-shot chain-of-thought instruction that asks for multi-label exploration performs no better than plain zero-shot CoT.","The approach trades in-distribution accuracy and token cost for OOD robustness, a trade-off the paper explicitly flags as an open problem.","Diversity in the reasoning space yields larger OOD gains than diversity in the demonstration space, based on the comparison with retrieval-based example selection."],"supporting_citations":[{"why":"Introduced X-ICL, the single-explanation baseline that X2-ICL extends, and the meta-prompt procedure for generating explanation-augmented demonstrations.","marker":"He et al., 2024"},{"why":"Defined in-context learning, the paradigm whose OOD robustness the paper aims to improve.","marker":"Brown et al., 2020"},{"why":"Self-consistency, the stochastic multi-path baseline that the ablation compares against systematic reasoning exploration.","marker":"Wang et al., 2023c"},{"why":"Zero-shot chain-of-thought, the baseline in the ablation that tests whether explicit demonstrations of label-conditioned reasoning are necessary.","marker":"Kojima et al., 2022"},{"why":"HANS, one of the eight OOD datasets used in the main evaluation.","marker":"McCoy et al., 2019"},{"why":"ANLI, the adversarial NLI dataset used for the main results and the ablation study.","marker":"Nie et al., 2020a"},{"why":"PAWS, the OOD benchmark used for paraphrase identification.","marker":"Zhang et al., 2019"},{"why":"Non-explanation-based robustness method (random labels and task descriptions) that the appendix compares against explanation-based approaches.","marker":"Sun et al., 2024"}],"fun_headline_variants":["Reasoning for every label boosts out-of-distribution robustness","X2-ICL: exploring all label reasons outperforms standard ICL","All-label explanations deliver robust in-context learning","Covering all label reasoning beats baseline ICL","Explaining all labels improves ICL robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the explanations a model generates for labels it never saw in the demonstrations are informative, and that the final step actually compares their validity rather than picking a confident-sounding story; Section 4.5 documents exactly such failure cases.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning for every label boosts out-of-distribution robustness","X2-ICL: exploring all label reasons outperforms standard ICL","All-label explanations deliver robust in-context learning","Covering all label reasoning beats baseline ICL","Explaining all labels improves ICL robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1278,"prompt_tokens":857,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":473,"tokens_out":421,"duration_ms":3881,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:25:22.440184+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test whether the comparison step, rather than the extra text, causes the gains by taking the X2-ICL prompt and randomizing or deleting the 'Possible Reasoning' blocks at test time while keeping all other content identical; if OOD accuracy stays roughly the same, the claimed validity-comparison mechanism is not what drives the improvement.","supporting_citations":[],"review_version":1}