{"id":"e79ea7e2-90e3-4c81-b5f9-5692e54af675","arxiv_id":"2507.23211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A retrieval method that uses semantically similar wrong answers as anchors to find corrective correct demonstrations improves few-shot in-context learning accuracy on seven reasoning datasets.","lead":"Large language models answer questions better when shown a few examples, and this paper uses examples the model got wrong to help choose better correct examples. The method improves accuracy on several reasoning benchmarks compared with picking only the most similar correct examples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: two-hop retrieval with negative anchors is not compared against two-hop retrieval with non-negative anchors, so the gains in Table 1 may not be caused by negative-sample information.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the paper never shows that the negative label, rather than the second retrieval hop, causes the improvement. My proposed positive-anchor control sharpens the reader's suggested random-anchor control: both anchors are equally query-similar, so the only difference is whether the anchor is a model error or a correct example. The absence of error bars and significance tests is real but secondary; it would not by itself distinguish the mechanism. The post-hoc selection of n in Table 2 and the missing code/data also weaken the paper, but they do not change the verdict. The central claim is plausible and the reported numbers are directionally consistent with it, so rejection is not warranted. The paper should remain CONDITIONAL until the anchor-type ablation and significance testing are provided.","tokens_in":10651,"tokens_out":8028,"duration_ms":97233,"concrete_test":"Run an anchor-type ablation on all seven datasets with the same corpus split, Sentence-BERT embeddings, and GPT-3.5-Turbo, for k=2 and at least five seeds: (1) Ours(1 pos-new) exactly as in Table 1; (2) the same two-hop retrieval, but the second-hop anchor is the top-1 positive retrieved directly for q, with the anchor itself excluded from the candidate set; (3) Ours(2 pos-new) exactly as in Table 1; (4) the same two-hop retrieval, but the anchors are the top-2 positives retrieved directly for q, again excluding each anchor from its own candidate set. Report mean plus/minus standard deviation for all arms. If arm (2) falls within one standard error of arm (1), and arm (4) within one standard error of arm (3), on a majority of the seven datasets, the negative-sample information is not load-bearing and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of §2.2 is that a negative example retrieved for a query q is a semantic pointer to q's failure mode, so the positive example nearest to that negative is an implicit error correction. What the experiments actually compare is a one-hop retriever (Similarity Few-Shot) against a two-hop retriever whose intermediate anchors are drawn from the negative corpus. No ablation removes the negative label while keeping the two-hop architecture. Because the negative anchors are themselves selected by similarity to q, the second hop may simply be a diversity-inducing retrieval step. Table 2 is consistent with this: on CommonsenseQA, the n=6 condition with no query-similar positive at all reaches 80.0, suggesting the anchor type matters less than the second hop. Also, no error bars or significance tests are reported, so small deltas such as 75.9 vs 75.6 (CommonsenseQA, Table 1) are not interpretable. The decisive experiment is an anchor-type ablation: keep two-hop retrieval but replace negative anchors with equally query-similar positive anchors. If that control matches the proposed method, the central claim that error information in negative samples drives the gain is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage method for few-shot in-context learning in which negative samples (incorrectly answered examples) are used as semantic anchors to retrieve additional positive examples. The authors first build positive and negative corpora by running Zero-Shot-CoT on a clustered subset of the training data, then at inference retrieve k/2 query-similar positives and k/2 query-similar negatives, and for each negative retrieve the most similar positive from the positive corpus. The final prompt concatenates the query-similar positives and the negatively-anchored positives. Experiments on seven datasets over arithmetic, commonsense, and symbolic reasoning with GPT-3.5-Turbo are reported in Tables 1 and 2, with the stated conclusion that negative samples improve demonstration selection.","tokens_in":10838,"tokens_out":3958,"duration_ms":43580,"significance":"If the central claim is correct, the paper offers a simple, API-compatible method for improving few-shot ICL without additional training or stronger models, and it would be one of the few demonstrations that incorrect examples carry usable corrective signal for demonstration selection. The manuscript is clearly written, the method is easy to reproduce from the description, and Table 2 makes an initial attempt to study the number of negative anchors. However, the empirical support is currently suggestive rather than conclusive: the central mechanism is not isolated by any control, the headline 'consistently superior' claim is contradicted by several reported numbers, and no variance estimates are given. The significance is therefore conditional on the missing control experiments being added and the claims being appropriately qualified.","major_comments":[{"comment":"The central claim that negative-sample information drives the gains is not isolated by the experimental design. The proposed method is a two-hop retriever whose intermediate anchors are negative examples, but no control replaces those anchors with equally query-similar positive examples while keeping the two-hop architecture. If such a control matches Ours(2 pos-new), the observed improvements could be caused by the second retrieval hop or by increased demonstration diversity rather than by error-related information in negative samples. The m=0, n=6 row in Table 2 (e.g., 80.0 on CommonsenseQA) makes this concern concrete, because it removes query-similar positives entirely and still performs well.","section":"§2.2 and Tables 1–2"},{"comment":"The statement that the proposed approach 'consistently achieves superior performance compared to all baseline methods across all seven datasets' is contradicted by the reported numbers: Ours(1 pos-new) is below Random Few-Shot on CommonsenseQA (75.2 vs 75.7) and below Similarity Few-Shot on CommonsenseQA (75.2 vs 75.6); Ours(2 pos-new) is below Random Few-Shot on AddSub (83.6 vs 85.5) and on GSM8K (77.4 vs 77.7). The claim should be restricted to the best variant per dataset and properly qualified.","section":"§3.2, Table 1"},{"comment":"No variance estimates, repeated runs, or significance tests are reported anywhere. Many of the differences are small in absolute terms (e.g., 75.9 vs 75.6 on CommonsenseQA, 83.6 vs 83.0 on AddSub in Table 1), so without error bars it is not possible to determine whether the gains are reliable. This is load-bearing because the paper's headline conclusion is an empirical superiority claim.","section":"§3.2, Tables 1 and 2"},{"comment":"The analysis in Table 2 conflates the number of negative anchors with the composition of the final demonstration set. In rows such as m=0, n=6, the demonstrations contain only second-hop positives and no query-similar first-hop positives, so comparisons across rows change both the anchor type and the final prompt composition. The interpretation that 'using too many negative examples leads to a performance decline' is therefore not directly supported by the table, because the n=6 condition also removes all first-hop query-similar positives.","section":"§3.2, Table 2"}],"minor_comments":[{"comment":"There are several typographical errors that should be fixed: 'Similarty-Based Few-Shot CoT' should be 'Similarity-Based Few-Shot CoT', Table 2's header 'ComonSenseQA' should be 'CommonsenseQA', and Figure 2 contains 'Postive Example' and 'Tset Question'.","section":"§3.1"},{"comment":"Figure 1 and the surrounding text contain garbled or illegible character sequences that appear to be a rendering artifact; these should be repaired before publication.","section":"Figure 1"},{"comment":"References [3] and [4] are the same arXiv paper (2310.20689) listed twice; they should be consolidated.","section":"References"},{"comment":"The choices of cluster counts (4, 6, 8, 10 for different datasets) and the fixed k=2 are introduced without justification; a brief sentence on why these values were chosen, or a sensitivity check, would strengthen the presentation.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is modest and the empirical support is currently insufficient for the strong claim in the abstract. The decisive missing experiment is the two-hop positive-anchor control; without it, the proposed mechanism is not distinguishable from a generic diversity-inducing second retrieval hop. If the authors add that control, report variance or significance, and qualify the 'consistently superior' claim, the paper could become acceptable. I would also suggest pruning the self-citations [13,14] from the introduction, as they are not central to the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely new: retrieve a negative example (a model error) for the query, then use it as a semantic anchor to retrieve a positive example that corrects it. Prior learn-from-mistakes work uses negative data for fine-tuning or self-consistency, not for anchor-based demonstration retrieval. The method is clearly described, and the clustered corpus construction is sensible. The paper evaluates on seven datasets over three reasoning types, with reasonable baselines including a contrastive variant. The worked example in Figure 2 makes the mechanism easy to follow. Related work is fair.\n\nThe soft spot is the central empirical claim. The comparison is one-hop positive retrieval versus two-hop retrieval where the intermediate anchors are negative. There is no ablation that keeps two-hop retrieval but replaces negative anchors with equally query-similar positive anchors. Without that control, the gains in Table 1 could come from the second hop alone, not from negative-sample information. Table 2 actually supports that alternative reading: on CommonsenseQA, the condition with zero query-similar positives (m=0, n=6) reaches 80.0, suggesting anchor type is less important than the two-hop structure. The paper's own 'consistently superior' wording is contradicted by Table 1, where Ours(1 pos-new) is below Random on CommonsenseQA and Ours(2 pos-new) is below Random on AddSub and GSM8K. There are no error bars or significance tests, so small deltas like 75.9 vs. 75.6 are not interpretable. The choice of m and n in Table 2 is post-hoc, without hold-out validation. Minor: the text after Figure 1 is garbled, and no code or data is supplied.\n\nThe paper deserves a serious referee because the idea is original and the framing is clear, but the current evidence is suggestive rather than conclusive. A referee should ask for the anchor-type ablation, significance testing, and a pre-specified m/n choice. If the control matches the proposed method, the central claim is unsupported; if the control does not, the method still has value as a practical retrieval trick. I would not cite it as established until that control is in.","headline":"Genuinely new idea—negative examples as anchors for positive demonstration retrieval—but the experiments miss the control that would prove the mechanism, so treat the empirical claim as suggestive, not established.","tokens_in":706,"tokens_out":1458,"would_cite":false,"duration_ms":30217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the examples a model gets wrong to choose the correct ones improves few-shot in-context learning.","keywords":["few-shot in-context learning","negative samples","demonstration retrieval","semantic similarity","chain-of-thought","prompt engineering","error correction","reasoning tasks"],"falsifier":"Run the same two-hop retrieval with each negative anchor replaced by a random incorrect example of equal similarity to the query; if accuracy does not drop, the effect is due to an extra retrieval hop rather than to the negative sample's content. A second check is to compare against a single-hop baseline that retrieves the same number of positives directly by query similarity at a larger $k$; if that matches the two-hop results, the negative anchor adds nothing beyond more demonstration slots.","tokens_in":10380,"feed_emoji":"🎯","tokens_out":7687,"duration_ms":78949,"temperature":0.7,"pith_summary":"This paper tries to establish that the examples a language model answers incorrectly are not wasted context: they can be used to choose the correct examples that make up a few-shot prompt. The proposed method splits a dataset into positive and negative corpora by running zero-shot chain-of-thought over half the data, then for each test question retrieves similar examples from both corpora. It then treats each retrieved negative example as an anchor and fetches the most similar positive example, concatenating these newly retrieved positives with the query-similar positives as the final demonstrations. Across seven datasets spanning arithmetic, commonsense, and symbolic reasoning, this construction reports higher accuracy than using only the most similar positive examples. The paper's claim is that the wrong answer closest to a question carries information about the specific error the question invites, and that the correct example closest to that wrong answer supplies the corresponding correction.","feed_headline":"Failed answers help choose examples that lift few-shot accuracy","feed_subtitle":"Retrieving correct examples anchored on the model's own wrong answers beats similarity-only prompts across seven datasets.","key_machinery":"The central mechanism is two-hop semantic retrieval with negative samples as anchors. A negative sample is an example the model answered incorrectly, determined by running Zero-Shot-CoT on half of each k-means cluster and comparing the prediction with the gold answer; the rest of the pipeline uses Sentence-BERT embeddings for similarity. For a query, the first hop selects $k/2$ examples from the positive corpus and $k/2$ from the negative corpus by embedding distance. The second hop takes each selected negative example and retrieves the single most similar positive example from the positive corpus, on the view that the correct answer to a near-miss question is an implicit error-correction exercise. The final demonstration concatenates the first-hop positives and the negative-anchored positives.","core_discovery":"The paper claims that a demonstration constructed by a two-hop similarity search over a corpus of answered examples outperforms demonstrations built only from the most similar correct examples. In the first hop, the query retrieves its nearest positive and negative examples; in the second hop, each negative example retrieves its own nearest positive from the positive corpus. The final prompt contains the first-hop positives plus these new negative-anchored positives. The best variant, which retrieves two negatives and then two new positives anchored on them, scores 84.1 on SVAMP, 75.9 on CommonsenseQA, 67.2 on StrategyQA, and 83.3 on Last Letter, all above the corresponding similarity-few-shot baselines, and the mixed one-positive-one-new-positive variant beats the baseline on the three arithmetic datasets and on Last Letter. The authors interpret this as evidence that negative samples improve in-context learning by improving positive-sample selection rather than by adding in-context contrastive warnings.","pith_inferences":["A control the paper does not run would test whether the improvement comes specifically from negative information rather than from the extra retrieval step: replace each negative anchor with a randomly chosen wrong example, or with a positive anchor, and compare accuracy.","If error types cluster, then the second hop should retrieve positives that fix a specific failure; one testable consequence is that clustering the negative corpus by predicted-error category and requiring retrieval to stay within the same category would tighten the corrections and might further lift accuracy.","The authors' 'semantic drift' explanation for too many negatives predicts a monotone decline in the relevance of second-hop positives as the number of anchors grows, which could be measured directly with embedding distance between anchors and retrieved positives.","A further extension would apply the same negative-anchored retrieval to larger demonstration counts and to open-source models, to see whether the gains persist outside the single model and $k=2$ setting used here."],"forward_implications":["If the claim holds, negative examples are informational assets for demonstration construction, so future few-shot retrievers can mine both correct and incorrect corpora instead of ignoring failures.","The task-dependent optimum found in the paper suggests that arithmetic and symbolic reasoning want a small dose of error-anchored positives, whereas commonsense reasoning works best when every demonstration is an error-anchored positive.","Since the method only needs zero-shot inference and a sentence embedding model, any unlabeled task set can be turned into positive and negative corpora without human annotation.","The two-stage design removes the need for an explicit error-correction step from a stronger model; retrieval itself supplies the correction."],"supporting_citations":[{"why":"Supplies Zero-Shot-CoT, the technique that produces predictions on the training split and thereby labels examples as positive or negative for the two corpora.","marker":"[9]"},{"why":"Supplies Sentence-BERT, the embedding model used for all semantic-similarity retrieval in both hops.","marker":"[20]"},{"why":"Provides the finding that models trained on positive and negative samples solve only partially overlapping cases, the evidence the paper cites for treating negative samples as carrying extra information.","marker":"[12]"},{"why":"Establishes that learning from mistakes can improve reasoning, the line of work the paper extends while avoiding explicit external error correction.","marker":"[3]"}],"fun_headline_variants":["Using wrong answers to pick better examples for few-shot learning","Negative samples improve in-context learning by refining example selection","Two-hop retrieval with negative anchors beats similarity-only prompts","Wrong answers boost few-shot accuracy by guiding positive example choice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a wrong example that is semantically close to the query is close because it shares the query's specific error pattern, and that the correct example nearest to that wrong example therefore teaches the model how to fix that specific mistake.","fun_headline_variants_meta":{"raw":{"variants":["Using wrong answers to pick better examples for few-shot learning","Negative samples improve in-context learning by refining example selection","Two-hop retrieval with negative anchors beats similarity-only prompts","Wrong answers boost few-shot accuracy by guiding positive example choice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2696,"prompt_tokens":956,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1674}},"tokens_in":572,"tokens_out":1740,"duration_ms":13833,"temperature":1.0,"reasoning_tokens":1674,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:57:14.688783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-hop retrieval with each negative anchor replaced by a random incorrect example of equal similarity to the query; if accuracy does not drop, the effect is due to an extra retrieval hop rather than to the negative sample's content. A second check is to compare against a single-hop baseline that retrieves the same number of positives directly by query similarity at a larger $k$; if that matches the two-hop results, the negative anchor adds nothing beyond more demonstration slots.","supporting_citations":[{"cited_title":"Advances in neural information processing systems 35, 22199–22213 (2022) Title Suppressed Due to Excessive Length 9","cited_arxiv_id":null,"evidence_quote":"Supplies Zero-Shot-CoT, the technique that produces predictions on the training split and thereby labels examples as positive or negative for the two corpora."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Provides the finding that models trained on positive and negative samples solve only partially overlapping cases, the evidence the paper cites for treating negative samples as carrying extra information."}],"review_version":1}