{"id":"72480f8d-56ad-41a6-89ff-aafcd86d3e49","arxiv_id":"2506.00418","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dual-debiasing method normalizes perplexity by the model's prior knowledge and a query-specific baseline, detecting noisy ICL demonstrations even at 80% noise.","lead":"The authors propose a way to automatically spot bad or mislabeled examples before using them as few-shot examples for a text-generating AI. Their method compares each answer against random answers to the same question, and it keeps performance close to using only correct examples, even when 80% of the examples are wrong.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The discrete neighbor approximation in Eq. 9-10 estimates a query-conditional average, not the joint neighborhood average of Eq. 7; the claimed cross-domain comparability of the Sample Cleanliness Score is unproven and likely corpus-dependent.","rationale":"The reader's weakest assumption (Eq. 7-10) is the right target. The paper's headline 'absolute sample cleanliness' depends on the extrinsic-bias estimator being a valid domain normalization, but the discrete estimator conditions on the query and is therefore not the neighborhood average defined in Eq. 7. The paper does not prove that the resulting score is comparable across domains or invariant to corpus composition; its own Limitations section admits the neighbor sampling is load-bearing. The empirical claim of clean-comparable ICL is additionally weakened by reporting the best of Cin/Cout, but that is a reporting flaw that can be fixed. The proposed check directly varies the neighbor corpus (composition and length) and measures AUC/rank stability, which would settle whether the 'absolute' claim is a property of the metric or an artifact of the chosen corpus. If the check passes at the tested settings, the conditional verdict could be upgraded; if it fails, the central claim needs qualification. For now, CONDITIONAL remains the right verdict.","tokens_in":16229,"tokens_out":9646,"duration_ms":93877,"concrete_test":"Recompute the Sample Cleanliness Score for a fixed set of clean/noisy demonstration pairs from NQ and WebQ using three variants of the neighbor corpus C: (i) the original external Cout, (ii) Cout with 80% of its annotations replaced by the noisy annotations from the training set, and (iii) a corpus restricted to annotations of length greater than 7 tokens. For each variant, compute the AUC for clean/noisy discrimination and the rank correlation of I values across variants. If the AUC changes by more than 0.05 or the rank correlation drops below 0.9, the metric is corpus-relative and the 'absolute cleanliness regardless of noise level' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the Sample Cleanliness Score 'uncovers absolute sample cleanliness regardless of the overall corpus noise level' rests on the extrinsic-debiasing step. Equation 7 defines the extrinsic bias Phi((x,y)) as the average of L_de-int over a continuous neighborhood N((x,y)) = {(x',y'): d_edit(y,y') < eta}, using a distance that ignores x. The tractable estimator in Eq. 10 replaces this with (1/N) sum_z L_de-int(y'_z | x), sampling y'_z from a corpus C while fixing the original query x. This is a query-conditional average E_{y'|x}[L_de-int(y'|x)], not an estimate of the joint neighborhood average. Since the paper's own premise is that the bias varies with query domain, conditioning on x cannot remove that bias. The claimed cross-domain comparability, I(x1,y1) > I(x2,y2) for clean vs noisy, requires that B(x)/L_de-int(y|x) separates clean from noisy uniformly, where B(x) is the corpus-dependent baseline. No argument establishes this; the paper's Limitations section concedes the metric 'relies heavily on the quality and diversity of the neighbor samples.' The main results further obscure the dependence by reporting the best of two corpora (Cin/Cout) without a selection rule. A concrete test is to vary the composition of C while holding the evaluated demonstrations fixed; if the score's clean/noisy separation changes, the 'absolute' claim fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies noisy annotations in in-context learning (ICL) for text generation. It proposes a Sample Cleanliness Score built from per-token negative log-likelihood after two debiasing steps: an intrinsic step that normalizes L(y|x) by the annotation-only loss L(y), and an extrinsic step that divides by a neighborhood average of the intrinsic-debiased loss. The score is used with a two-component Gaussian mixture model to split the training set into clean and noisy demonstrations; noisy samples are discarded and ICL is performed on the remaining set. The paper evaluates the method on NQ, WebQ, SciQ, and SQuAD, with three retrievers, relevant/irrelevant noise at ratios up to 0.8, and several metric LLMs, reporting performance close to a fully clean corpus and superior to baselines including local perplexity ranking.","tokens_in":16601,"tokens_out":6964,"duration_ms":65584,"significance":"The empirical contribution is substantial if the central claim holds: the method is evaluated broadly and shows consistent gains over Naive ICL, Random deletion, and LPR, with sensitivity analyses for gamma and N_neighbor, a robustness study across metric LLMs, and a failure-case analysis. The use of fixed default hyperparameters and an unsupervised GMM avoids fitting the thresholds to test labels. However, the 'absolute' cross-domain comparability of the score is the load-bearing theoretical claim, and it is neither proven nor convincingly supported by the discrete estimator; the paper itself states that the metric depends on the quality and diversity of neighbor samples. The reported best-of-Cin/Cout selection also requires a practical selection rule before the method can be used as claimed.","major_comments":[{"comment":"The discrete estimator in Eq. (10) does not estimate the continuous quantity in Eq. (7). Equation (7) averages Lde-int over the joint neighborhood N((x,y))={(x',y'): d_edit(y,y')I(x2,y2) whenever y1 is clean and y2 is noisy, even across domains, is stated without proof. The within-query inequality in Eq. (5) only supports comparisons for a fixed query; it does not imply that the ratio Phi(x,y)/Lde-int(y|x) separates clean from noisy uniformly after dividing by a corpus-dependent Phi. Either prove this inequality under explicit assumptions on the metric space and the sampling distribution of C, or collapse the claim to an empirical observation. As written, this unsupported inequality is the mathematical content of 'absolute sample cleanliness' and must be justified.","section":"Section 3.3, Eq. (8)"},{"comment":"The paper reports the main results using 'the optimal performance achieved by either Cin or Cout' without specifying how that choice is made. If the better corpus is selected using the test split, the numbers in Table 1 are optimistic and the pipeline as described is not reproducible. A practical method needs a fixed default corpus or a validation-based selection rule; at minimum, results for Cin and Cout should be reported separately in the main table instead of the best of the two.","section":"Section 4.1, Implementation details"},{"comment":"The Limitations paragraph concedes that the method's effectiveness 'relies heavily on the quality and diversity of the neighbor samples' and may be degraded in domains with limited data. This directly conflicts with the Abstract's claim that the metric 'uncovers absolute sample cleanliness regardless of the overall corpus noise level.' Since Eq. (10) averages over annotations sampled from C, a corpus dominated by noisy annotations contributes noisy values to Phi, making the score dependent on the noise composition of C. The paper should either qualify the absolute claim or provide an experiment that varies the clean/noisy composition of C while holding the scored demonstrations fixed and shows that clean/noisy separation is unchanged.","section":"Limitations"}],"minor_comments":[{"comment":"The text says 'serve as alternation for the domain' where it should be 'alternative'.","section":"Section 3.3, Eq. (9)"},{"comment":"The caption for Figure 2d reads 'Band vs Number on GPT-Neo-2.7B', while the text in Section 3.3 says a similar issue is observed for Music-Band vs Number under Gemma-2b; the caption and body text disagree.","section":"Figure 2d"},{"comment":"The sentence 'the efficient approach by Liu et al. that extracts in-context vectors' omits the year and venue of the citation; please supply the full reference or use the numbered citation format consistently.","section":"Related Work"},{"comment":"The comment '/* Metric Calculation For the Whole Training Set */' appears twice in Algorithm 1; the second occurrence, before the ICL inference step, should be replaced with a description of the inference phase.","section":"Algorithm 1"},{"comment":"The hyperparameter gmm_part_thres is listed in Table 6 but is never defined or discussed in Section 3.4; please define it and explain how the per-dataset values were chosen.","section":"Appendix, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is worth a revision, but the central 'absolute' claim is currently stronger than the evidence. The gap between Eq. (7) and Eq. (10), the unsupported cross-domain inequality, and the best-of-two-corpora reporting are all load-bearing issues. The empirical results are promising, and I would be open to acceptance after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a genuinely new cleaning score for noisy ICL text generation, with surprisingly strong results at high noise, but the theoretical story is sloppier than the experiments, and the headline numbers are partly cherry-picked. Worth refereeing; I would not cite it yet.\n\nWhat is new: the Sample Cleanliness Score divides per-token NLL conditioned on the query by the unconditional NLL (intrinsic debiasing), then normalizes by a neighbor-based average (extrinsic debiasing). That dual-debiasing combination is not in Gao et al.'s LPR or Kang et al.'s Rectification. The experiments are admirably broad—four QA datasets, three retrievers, noise up to 0.8, several metric LLMs—and the gains over LPR at 0.6–0.8 noise are large and consistent. The sensitivity analyses for gamma and N_neighbor suggest those defaults are not fitted to the test set. The Limitations section is honest, conceding the metric depends on the quality and diversity of neighbors.\n\nThe soft spots are real. First, the main theoretical gap: Eq. 7 defines the extrinsic bias as an average over a neighborhood using an edit distance that ignores the query x, but the tractable estimator in Eq. 10 fixes x and averages Lde-int(y'|x) over sampled y'. That is a query-conditional average, not a joint neighborhood average. The paper's formal claim that I(x1,y1) > I(x2,y2) across domains follows from this is unsupported. The estimator might actually be the right normalization—if the bias is about query domain, conditioning on x makes sense—but then Eq. 7 should say so. Fixable, but not cosmetic.\n\nSecond, the abstract's \"absolute cleanliness regardless of corpus noise\" overstates: the score depends on the neighbor corpus C, as the Limitations admit. The paper does not test how varying C affects clean/noisy separation for fixed demonstrations, which is the obvious falsification experiment.\n\nThird, the main results report the best of two corpora (Cin/Cout) with no selection rule. That is cherry-picking; the baselines do not get the same courtesy. The analysis tables use Cout only, so the pattern is there, but the headline table is not a fair comparison.\n\nFourth, no code, no error bars. Seeds are averaged but variance is not reported. For an empirical paper of this type, that matters.\n\nNone of this kills the work. The dual-debiasing idea is genuinely new, and the empirical signal is strong enough that the core result may survive a careful re-run. But a serious referee should demand: fix the math or retract the proof; report both corpora separately or give a fixed selection rule; and release code. I would send it to review with those demands, not desk-reject it.","headline":"Novel, empirically strong cleaning score for noisy ICL, but the theory underpinning the 'absolute' claim is loose and the headline results are cherry-picked between two corpora; deserves review with major revision demands.","tokens_in":17112,"tokens_out":3999,"would_cite":false,"duration_ms":39434,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Sample Cleanliness Score, computed by dividing conditional per-token NLL by unconditional NLL and by a neighbor-averaged domain-bias estimate, separates clean from noisy demonstrations in text-generation in-context…","keywords":["noisy in-context learning","sample cleanliness score","dual debiasing","perplexity","text generation","label noise","demonstration selection","LLM prior knowledge"],"falsifier":"Take a corpus with 80 percent noisy annotations where the noisy annotations are deliberately short and plausible, such as swapping in other valid answers from the same domain, compute the Sample Cleanliness Score, and check whether the two-component mixture still recovers the true clean subset with the AUC gains reported; the claim would fail if the neighbor average is dominated by noisy annotations and clean samples from hard domains receive scores overlapping the noisy component.","tokens_in":15986,"feed_emoji":"🧹","tokens_out":5247,"duration_ms":46002,"temperature":0.7,"pith_summary":"The paper argues that perplexity-based noise detection in in-context learning fails at high noise ratios because raw per-token negative log-likelihood is biased twice: the LLM is familiar with the annotation text itself, and the LLM has uneven knowledge across domains. It proposes a Sample Cleanliness Score that removes both biases, dividing the conditional NLL by the unconditional NLL and then by a neighbor-averaged estimate of domain difficulty, so a clean demonstration scores higher than a noisy one whether or not they come from the same topic. The claim is that this score detects noisy demonstrations reliably even when 80 percent of the corpus is noisy, and that in-context learning using only the detected clean subset performs about as well as using a fully clean corpus. A reader should care because noisy annotations are common in real data, and existing methods assume noise is a minority.","feed_headline":"Dual debiasing keeps ICL accurate at 80% noise","feed_subtitle":"A Sample Cleanliness Score removes annotation and domain bias from perplexity, matching clean-corpus results.","key_machinery":"The Sample Cleanliness Score I(x,y)=Phi((x,y))/L_de-int(y|x), where L_de-int(y|x)=L(y|x)/L(y) is the per-token NLL of the annotation given the query divided by the per-token NLL of the annotation alone, and Phi is a neighbor-based estimate of the domain-level average of L_de-int over pairs with the same query and random annotations from a corpus. The mechanism is a ratio: high conditional loss caused by genuine mismatch survives, while high conditional loss caused by a hard domain or by the annotation's own improbability is divided out, making scores comparable across queries, annotations, and knowledge domains.","core_discovery":"The central discovery is a two-step debiasing identity for per-token NLL. First, intrinsic debiasing defines L_de-int(y|x)=L(y|x)/L(y), the conditional per-token loss divided by the unconditional per-token loss of the annotation alone, which removes the LLM's prior familiarity with the annotation string. Second, extrinsic debiasing divides L_de-int by the average of L_de-int over a neighborhood of synthesized pairs that keep the query x but replace the annotation with random annotations sampled from a corpus, estimated by edit-distance-bounded neighbors. The Sample Cleanliness Score I(x,y) is then compared across all demonstrations; a two-component Gaussian mixture model on the scores separates clean from noisy, and discarding the noisy subset before retrieval yields in-context-learning performance the paper reports as comparable to a fully clean demonstration set, across four text-generation datasets, two noise types, three retrievers, and noise ratios up to 0.8.","pith_inferences":["The paper leaves implicit that the same dual-debiasing ratio could be applied to other per-sample loss-based selection problems, such as filtering instruction-tuning data or selecting demonstrations for a specialized domain, where annotation priors and domain skew play comparable roles.","The neighbor-based estimate reuses the same query with random corpus annotations; an extension the paper does not explore is replacing corpus-sampled annotations with LLM-generated or paraphrased ones, which could change the domain-bias estimate for long or rare annotations.","The paper's own failure analysis ties detection errors to long annotations; a direct extension would be a length-normalized or length-binned version of the score, since the extrinsic-bias approximation degrades when annotation lengths vary widely.","Because the score measures the metric model's perception, a corpus scored by one LLM may not transfer directly to another; a testable corollary is whether a calibration step between metric models is needed before deploying the score on a new inference model."],"forward_implications":["Noise-ratio independence: because the score is normalized per sample rather than ranked within the corpus, clean samples sit on the same side of the mixture split even when noisy samples are the majority; experiments show stability up to 0.8 noise.","Deletion instead of repair: once noisy demonstrations are identified, simply discarding them is enough to match clean-corpus in-context-learning performance, removing the need for replacement or correction steps.","Model-scale robustness: the same score works when computed with a 1.3B model as with a 7B model, with only negligible in-context-learning performance decline, so noise screening can be done cheaply.","Retriever independence: the gain holds across Random, TopK, and DPP retrievers, meaning the cleanliness signal is about the demonstration set, not a particular retrieval strategy."],"supporting_citations":[{"why":"Supplies the LPR baseline and the noise-generation procedure for relevant and irrelevant noise that the paper compares against and extends.","marker":"(Gao et al., 2024)"},{"why":"Llama-2-7B is the default inference and metric LLM throughout the experiments.","marker":"(Touvron et al., 2023)"},{"why":"The OpenICL framework implements the retrieval and inference pipeline used for all methods.","marker":"(Wu et al., 2023)"},{"why":"The NQ dataset supplies the topic-based analysis that motivates extrinsic debiasing, and is one of the four evaluation datasets.","marker":"(Kwiatkowski et al., 2019)"},{"why":"GPT-Neo models are used as smaller or weaker metric LLMs to test the score's model-scale robustness.","marker":"(Black et al., 2021)"},{"why":"Gemma models are used as alternative metric LLMs to show the method transfers across architectures.","marker":"(Team et al., 2024)"},{"why":"Mistral-7B is used as a comparable-size stronger metric model in the LLM robustness analysis.","marker":"(Jiang et al., 2023)"},{"why":"TopK and DPP retrievers are the retrieval baselines used to show the method's independence of retriever choice.","marker":"(Ye et al., 2023)"}],"fun_headline_variants":["Dual debiasing purifies ICL at 80% noise","Debiased perplexity matching clean ICL under 80% noise","Sample Cleanliness Score: debias two ways, beat noisy ICL","Two-step debias for perplexity: robust to 80% noisy demos","ICL noise beaten by dual-debiased Sample Cleanliness Score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The score's cross-domain comparability depends on the assumption that the average of L_de-int over pairs formed by pairing each query with random annotations from a corpus faithfully captures the LLM's domain-level difficulty; if the corpus is dominated by noisy annotations or annotation lengths vary widely, that neighborhood average can be contaminated and the central claim of absolute cleanliness would break.","fun_headline_variants_meta":{"raw":{"variants":["Dual debiasing purifies ICL at 80% noise","Debiased perplexity matching clean ICL under 80% noise","Sample Cleanliness Score: debias two ways, beat noisy ICL","Two-step debias for perplexity: robust to 80% noisy demos","ICL noise beaten by dual-debiased Sample Cleanliness Score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1613,"prompt_tokens":906,"completion_tokens":707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":522,"tokens_out":707,"duration_ms":7164,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:05:35.512540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus with 80 percent noisy annotations where the noisy annotations are deliberately short and plausible, such as swapping in other valid answers from the same domain, compute the Sample Cleanliness Score, and check whether the two-component mixture still recovers the true clean subset with the AUC gains reported; the claim would fail if the neighbor average is dominated by noisy annotations and clean samples from hard domains receive scores overlapping the noisy component.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LPR baseline and the noise-generation procedure for relevant and irrelevant noise that the paper compares against and extends."}],"review_version":1}