Pith. sign in

REVIEW 5 cited by

Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.12685 v2 pith:MAPILGCW submitted 2022-05-25 cs.CL cs.AIcs.LG

Kang Min Yoo , Junyeob Kim , Hyuhng Joon Kim , Hyunsoo Cho , Hwiyeol Jo , Sang-Woo Lee , Sang-goo Lee , Taeuk Kim This is my paper
classification cs.CLcs.AIcs.LG
keywords ground-truthlearningin-contextdemonstrationsimpactinput-labellabelslabel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive. Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning. With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Discovering Latent Knowledge in Language Models Without Supervision

    cs.CL 2022-12 conditional novelty 8.0 of 10

    An unsupervised technique extracts latent yes-no knowledge from language model activations by locating a direction that satisfies logical consistency properties, outperforming zero-shot accuracy by 4% on average acros...

  2. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    cs.CL 2022-02 accept novelty 8.0 of 10

    Randomly replacing labels in in-context demonstrations barely hurts performance, showing that label space, input distribution, and sequence format drive in-context learning more than ground-truth labels.

  3. Towards Compute-Optimal Many-Shot In-Context Learning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.

  4. InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.

  5. The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences

    cs.CL 2025-09 unverdicted novelty 3.0 of 10

    The paper reduces a broad set of prompt engineering techniques to six core approaches and applies them to life sciences use cases while addressing common LLM pitfalls.

Pith tools