{"id":"91842d1c-5a9a-4cea-9f01-6ba89b5a8a68","arxiv_id":"2412.11923","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PICLe shows that pseudo-annotated, partially correct demonstrations can replace gold labels for in-context named entity detection, outperforming few-shot ICL with scarce gold examples on biomedical datasets.","lead":"The paper shows that in-context learning for named entity detection keeps working when the example annotations are only partly correct, as long as enough entity mentions survive. It then builds PICLe, a pipeline that uses machine-generated noisy labels instead of human labels and beats few-shot prompting with scarce gold examples on five biomedical datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PICLe never locates its actual pseudo-annotation operating point on the Section 4.2 perturbation curves; without that, the paper's transfer claim from 'partially correct labels help' to the 57.1 vs 52.8 result is not established.","rationale":"The load-bearing assertion is not merely that some partially correct demonstrations help; it is that the specific pseudo-annotations PICLe actually feeds into its ICL runs lie in that helpful regime. The paper's own Table 5 shows the pseudo-annotations are low-recall (37.5% after self-verification), and Figures 2/6 show deletion-style loss of correct mentions erodes downstream performance most quickly. The reader's weakest_assumption identifies this gap, and I agree. I would sharpen it by noting that Section 4.2's demonstration selection uses kNN retrieval, whereas PICLe uses random sampling within clusters (Sp-k-means), so the perturbation curves are not directly applicable to PICLe's operating point. The missing measurement is the recall/precision/entity-count of the actual demonstration sets used in Table 4. Without it, the 57.1 vs 52.8 advantage could be attributed to cluster diversity, merging, and self-verification rather than to tolerance of partially correct labels. This does not require rejecting the empirical result; the ablation study and multi-seed gold baselines provide partial support. But the condition should be made explicit and tested before the claim is accepted as fully supported. Hence I keep the CONDITIONAL verdict.","tokens_in":17745,"tokens_out":7400,"duration_ms":69727,"concrete_test":"Audit the actual PICLe demonstration sets: for each of the five datasets, for every Sp-k-means run and seed, compute the precision, recall, and F1 of the pseudo-annotated demonstrations against gold, plus the mean number of retained gold entities per demonstration, and overlay these points on the Figure 2/6 surfaces. If the operating points fall in the deletion-dominated low-recall region where the prediction curve reaches or falls below zero-shot, the Section 4.2 mechanism cannot explain PICLe's gain; as a control, also re-run inference with the gold labels on the same sampled demonstration sentences to isolate whether the benefit is noise tolerance or cluster diversity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference from Section 4.2 to PICLe has a quantification gap. The perturbation study establishes a beneficial regime using gold demonstrations corrupted by controlled whole-entity deletion/substitution and selected by kNN; the benefit is tied to retaining enough correct gold mentions and a semantic text-label mapping. PICLe's pseudo-annotations after self-verification have average recall 37.5% (Table 5): roughly 60% of gold mentions are absent. In Figures 2 and 6, deletion-heavy perturbations are exactly the ones whose downstream F1 decays fastest, falling toward the zero-shot level once entities are removed. The paper never reports the recall, precision, or number of surviving correct mentions of the actual demonstration sets sampled by Sp-k-means, nor whether those sets preserve the text-label correspondence that Section 4.1 identifies as essential. PICLe changes three variables at once (pseudo-label noise profile, random-within-cluster retrieval instead of kNN, and aggregation over K clusters), so the observed 57.1 vs 52.8 advantage could come from cluster diversity and self-verification rather than from the 'partially correct labels are as good as gold' mechanism. If a new domain's pseudo-annotations are mostly omissions with few surviving mentions, the claimed transfer benefit is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies which attributes of in-context demonstrations matter for low-resource named entity detection (NED) with LLMs. Through a perturbation study on gold demonstrations, the authors find that semantic input-label correspondence is essential but that partially correct labels can be as effective as fully correct labels, provided a sufficient number of entity mentions remain. Based on this, they propose PICLe, which builds a pool of pseudo-annotated samples via zero-shot LLM prediction, self-verification, and k-means clustering, then samples cluster-specific demonstrations (Sp-k-means) to predict entities and self-verifies the aggregated output. They evaluate on five biomedical NED datasets with three LLMs, reporting that PICLe outperforms zero-shot and gold-demonstration ICL with up to 100 gold examples. The paper includes ablations of each pipeline component and a model generalization study.","tokens_in":18063,"tokens_out":4761,"duration_ms":42401,"significance":"If the claimed result holds, PICLe would provide a practical way to reduce annotation cost for structured extraction tasks, and the perturbation analysis offers a useful counterpoint to prior work emphasizing the necessity of correct labels in ICL. The paper includes a component ablation (Table 3), an analysis of pseudo-annotation quality (Table 5), and multi-model/multi-dataset evaluation, which strengthen the empirical contribution. However, the central claim's support is currently incomplete: error bars are missing for PICLe, the connection between the perturbation curves and the actual pseudo-annotations is not quantified, and the comparison with gold ICL is confounded by the size of the demonstration pool. These issues are addressable but require additional experiments.","major_comments":[{"comment":"The paper never locates the actual pseudo-annotated demonstration sets on the perturbation curves of Figure 2. Table 5 reports that after self-verification the pseudo-annotation pool has an average recall of only 37.5%, which corresponds to a deletion-heavy regime in Figure 2 where downstream F1 decays most rapidly. Since Sp-k-means samples from this pool, the paper should report the precision, recall, and average number of correct gold mentions in the K demonstration sets actually used for inference and compare these values with the curves in Figure 2. Without such a comparison, the stated explanation of PICLe's benefit (that partially correct labels help ICL) is not verified for the actual operating point.","section":"§5 PICLe performance, Table 5 vs. §4.2 Figure 2"},{"comment":"PICLe results are presented without error bars or significance tests, despite the claim that PICLe 'significantly outperforms' zero-shot and gold ICL. The gold baselines show error bars over 5 seeds, and Section 5 states that PICLe experiments were also repeated over 5 seeds, so the variance should be reported for PICLe as well. The headline improvement of 57.1% vs. 52.8% is a macro-average over five datasets; per-seed results or a paired significance test are needed to support a 'significant outperformance' claim.","section":"§5 Figure 4 and Table 4"},{"comment":"The comparison between PICLe and gold ICL is confounded by the size of the demonstration pool: gold baselines sample from pools of 10, 50, or 100 labeled examples, while PICLe pools 1000 unlabeled (pseudo-annotated) training sentences. A control that uses the same 1000 sentences with gold labels (an oracle upper bound) would isolate the effect of label noise from the number and coverage of demonstration texts. Without this control, the observed advantage could come from larger demonstration diversity rather than from the method's tolerance to partial correctness, which would materially weaken the paper's central claim.","section":"§5 Experimental setup and Figure 4"}],"minor_comments":[{"comment":"The statement says 'We plan to share the code for PICLe and all of our experiments'; for a reproducibility statement in a methods-heavy paper, actual code release at submission would substantially strengthen the contribution.","section":"Appendix A Reproducibility statement"},{"comment":"The text says pseudo-annotated samples are clustered 'based off the embedding of their text and pseudo-annotations' but does not specify how the pseudo-annotations are embedded (e.g., as a concatenated string, or via a separate encoder). This should be clarified for reproducibility.","section":"§5, clustering description"},{"comment":"The column 'Ratio null samples (%)' is defined in the table note as samples with no labeled entities, but this definition should also appear in the caption for a self-contained table.","section":"Table 1 caption"},{"comment":"The phrase 'based off our findings' appears several times; the conventional phrasing is 'based on our findings'.","section":"Throughout"},{"comment":"The caption states that error bars show variance only for sampling subsets of gold examples; please state explicitly that PICLe results are reported without error bars (or add them) so readers can assess the comparison fairly.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the empirical direction is interesting, but the current evidence does not fully support the advertised claims. The missing operating-point measurement and the pool-size confound are central to the narrative that pseudo-annotations are as good as gold; these need additional experiments. The missing error bars for PICLe are also a standard but important issue for a claim of 'significant outperformance'. I recommend major revision rather than rejection because the concerns are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper has two halves. The first half, a perturbation study of in-context demonstrations for named entity detection, is the stronger contribution and genuinely new. The second half, the PICLe pipeline, works on average but the paper never quite joins it to the analysis. I'd referee it, and I'd ask for revisions.\n\nThe perturbation study is well done. Prior ICL analysis focused mostly on classification, where a label is either right or wrong. For NED, the authors examine partial correctness—deletions, substitutions, additions—and show that what matters is retaining enough correct entity mentions and a semantic text-label mapping. Fully incorrect labels hurt, but partially correct labels with enough surviving entities perform close to gold. That is a clean, useful result, and it transfers across the five biomedical datasets and the two open models they test.\n\nPICLe itself is a sensible combination of existing pieces: zero-shot pseudo-annotation, self-verification, k-means clustering of the pseudo-annotated pool, and cluster-specific ensembling. The ablation shows each component earns its keep; the averaged gain over gold ICL with 100 examples (57.1 vs 52.8) is plausible.\n\nThe soft spots are in proportion. First, the paper claims the perturbation findings motivate PICLe, but it never shows that PICLe's actual pseudo-annotations sit in the beneficial regime. After self-verification, pseudo-annotations have average recall of 37.5%—that's deletion-heavy—and the perturbation study shows deletion-heavy noise decays fastest. PICLe also changes three things at once (noise profile, retrieval, and ensembling), so the win could come from cluster diversity and self-verification rather than from the partial-correctness mechanism. This doesn't sink the method, but it means the narrative is overreaching. Second, reproducibility is weak: the code is \"planned\" and the headline table shows no variance for PICLe while showing variance for gold baselines. I can't tell if 57.1 vs 52.8 is significant. Third, the fine-tuning comparison shows BioMedBERT with 50 gold examples already beats all LLM baselines; that's a caveat worth putting in the abstract.\n\nWho should read it: anyone working on ICL for structured outputs, low-resource NER, or pseudo-labeling. The perturbation result alone is worth citing.\n\nMy recommendation: send to peer review, but ask for the operating-point analysis, code release, and seed-level variance. Fix those, and it's a solid paper.","headline":"The perturbation study is the real contribution; PICLe works on average, but the paper never proves its pseudo-annotations sit in the regime where partial correctness helps.","tokens_in":18565,"tokens_out":4899,"would_cite":true,"duration_ms":45550,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For in-context named entity detection, partially correct demonstration annotations transfer as effectively as gold ones, and a zero-human pipeline exploiting this beats few-shot ICL with 100 gold examples.","keywords":["in-context learning","named entity detection","pseudo-annotation","self-verification","low-resource NLP","biomedical NLP","demonstration selection","LLM prompting"],"falsifier":"Compute the mean number of correct entity mentions in the actual demonstration sets PICLe samples via Sp-k-means on a new dataset, and compare it to the threshold in the perturbation study at which downstream F1 falls to zero-shot levels for deletion-type noise; if the sampled sets fall below that threshold yet PICLe still claims gains, the central claim is falsified. Concretely, take a held-out dataset, run PICLe, and also run ICL with deletion-corrupted gold annotations matched to PICLe's demonstration F1; if deletion-corrupted ICL outperforms PICLe, the partial-correctness mechanism is not the operative one.","tokens_in":17566,"feed_emoji":"🧬","tokens_out":6365,"duration_ms":52564,"temperature":0.7,"pith_summary":"This paper asks what makes in-context demonstrations work for low-resource named entity detection, and reports a surprising answer: demonstrations whose entity annotations are only partially correct can transfer as effectively as fully correct gold annotations, as long as the label sets keep enough correct mentions and a semantic link to the input text. It establishes this with controlled perturbation studies that vary the type and degree of label corruption. Building on that finding, it introduces PICLe, a fully automatic pipeline that pseudo-annotates unlabeled sentences with a zero-shot LLM pass, self-verifies those annotations, clusters them, and samples cluster-specific demonstration sets for inference. On five biomedical NED datasets, PICLe with zero human annotation matches or outperforms standard ICL that uses up to 100 human-annotated demonstrations, suggesting that annotation effort can be redirected or eliminated in low-resource structured-output tasks.","feed_headline":"Automatic noisy demos beat 100 gold examples in NED","feed_subtitle":"Partially correct pseudo-annotations transfer as well as gold ones, making zero-human-annotation named entity detection possible.","key_machinery":"The central object is the demonstration perturbation curve of Section 4.2, which maps the precision, recall, and F1 of demonstration label sets to downstream prediction F1 and shows that performance is governed more by the number of surviving correct entity mentions and the text-label semantic correspondence than by the label set F1 itself. On the method side, the machinery is PICLe's pipeline: zero-shot pseudo-annotation, a self-verification step that filters wrong-type entities, k-means clustering of pseudo-annotated samples, and cluster-specific demonstration sampling (Sp-k-means) that produces diverse, specialized demonstration sets whose independent predictions are merged and self-verified at inference time.","core_discovery":"In-context learning for named entity detection is robust to noise in the demonstration labels, provided the demonstrations preserve a semantic correspondence between text and labels and retain a sufficient number of correct entity mentions. Through perturbation experiments that delete, substitute, or add entities with varying probabilities, the authors show that deletion-heavy noise degrades downstream F1 most sharply, while substitution and addition schemes keep prediction F1 close to gold-label ICL even at high perturbation rates; fully incorrect labels (random in-distribution, out-of-distribution, or swapped) drop performance below zero-shot. The paper's central claim is that this regime—partial correctness with adequate coverage—is exactly what zero-shot pseudo-annotations occupy, making it possible to replace gold demonstrations with automatically generated ones.","pith_inferences":["The deletion-sensitivity result implies that a practical improvement to PICLe would be to generate pseudo-annotations with higher recall (e.g., multiple zero-shot passes with diverse temperatures) before self-verification, since the perturbation study shows that the recall of the demonstration pool is what sustains downstream F1.","Because word-order corruption in demonstrations barely hurt performance, the paper's data are consistent with the hypothesis that NED demonstrations act primarily as format and entity-type priors rather than syntactic templates; this could be tested by shuffling word order in real PICLe demonstrations and measuring the F1 change.","The saturation of downstream F1 when switching from Mistral to GPT-3.5-Turbo pseudo-annotations suggests the inference-time aggregation (Sp-k-means plus self-verification) is the current bottleneck; varying the cluster count or the number of sampled demonstrations per cluster would directly test that."],"forward_implications":["If partial correctness suffices, low-resource NED no longer requires gold annotations for ICL; a practitioner can bootstrap from unlabeled text with zero human effort.","The perturbation results indicate that demonstration selection for structured-output tasks should maximize the number of correct entity mentions and preserve text-label correspondence, not maximize demonstration F1.","The finding that fully incorrect labels hurt NED (contrary to classification) implies that pseudo-annotation quality control should target the precision of surviving mentions, not just label-space coverage.","PICLe's gains hold across three LLMs and five biomedical datasets, suggesting the approach transfers across base models and entity types.","The comparison with fine-tuning suggests that when gold samples exceed roughly 50, supervised fine-tuning becomes more effective, so PICLe's niche is the truly annotation-scarce regime."],"supporting_citations":[{"why":"Establishes that in classification ICL demonstrations can transfer without correct labels; the paper's perturbation study contrasts NED with this finding.","marker":"Min et al., 2022"},{"why":"Quantifies that ground-truth labels matter in ICL, providing the baseline claim that PICLe's partial-correctness result extends and nuances.","marker":"Yoo et al., 2022"},{"why":"Z-ICL introduces pseudo-demonstrations for classification, the approach PICLe adapts to NED and compares against in ablations.","marker":"Lyu et al., 2023"},{"why":"Self-ICL is the most similar prior work generating zero-shot demonstrations for text classification, which PICLe extends to structured outputs.","marker":"Chen et al., 2023b"},{"why":"COSP's 10-run zero-shot pooling is the pseudo-annotation baseline PICLe compares against.","marker":"Wan et al., 2023a"},{"why":"PromptNER provides the entity-definition prompt formulation PICLe adopts for NED and self-verification.","marker":"Shen et al., 2023"},{"why":"Supplies the kNN demonstration retrieval method used throughout the perturbation studies.","marker":"Liu et al., 2022"},{"why":"Provides the self-verification method used for filtering pseudo-annotations and final predictions.","marker":"Weng et al., 2023"}],"fun_headline_variants":["Noisy demos match gold for in-context NED","PICLe: zero-annotation NED beats gold ICL","Partial label noise still yields strong NED ICL","Pseudo-annotations rival gold demos in NED","Robust ICL: noisy demos work for NED"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the noise in zero-shot pseudo-annotations (after self-verification) resembles the substitution or addition perturbations that keep enough correct mentions, not deletion-heavy noise; if a new domain yields pseudo-annotations with very few correct mentions, the transfer benefit should disappear.","fun_headline_variants_meta":{"raw":{"variants":["Noisy demos match gold for in-context NED","PICLe: zero-annotation NED beats gold ICL","Partial label noise still yields strong NED ICL","Pseudo-annotations rival gold demos in NED","Robust ICL: noisy demos work for NED"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1294,"prompt_tokens":925,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":541,"tokens_out":369,"duration_ms":4202,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:27:10.118509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the mean number of correct entity mentions in the actual demonstration sets PICLe samples via Sp-k-means on a new dataset, and compare it to the threshold in the perturbation study at which downstream F1 falls to zero-shot levels for deletion-type noise; if the sampled sets fall below that threshold yet PICLe still claims gains, the central claim is falsified. Concretely, take a held-out dataset, run PICLe, and also run ICL with deletion-corrupted gold annotations matched to PICLe's demonstration F1; if deletion-corrupted ICL outperforms PICLe, the partial-correctness mechanism is not the operative one.","supporting_citations":[],"review_version":1}