{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:LO2EULQNDEZ4R33CWWGIP7XQNV","short_pith_number":"pith:LO2EULQN","schema_version":"1.0","canonical_sha256":"5bb44a2e0d1933c8ef62b58c87fef06d4b52c2889253a1717819c66279b87b41","source":{"kind":"arxiv","id":"2202.12837","version":2},"attestation_state":"computed","paper":{"title":"Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Randomly replacing labels in in-context demonstrations barely hurts performance on classification and multiple-choice tasks across many models.","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Ari Holtzman, Hannaneh Hajishirzi, Luke Zettlemoyer, Mikel Artetxe, Mike Lewis, Sewon Min, Xinxi Lyu","submitted_at":"2022-02-25T17:25:19Z","abstract_excerpt":"Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2202.12837","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2022-02-25T17:25:19Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"274847321a77ccf35bc2559050cd585919998fa6fb270700a1bc1d6087907a26","abstract_canon_sha256":"e4913c3c1646b32d0c2b76710aa070a3c523fa645f2f2b1702d4594d1213ece1"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:52.846091Z","signature_b64":"y8KSO5K+4JV91FEaGp5J8yR+oD75stLRtJJsmj0BOzoYdLd9osaposp195/wGuO22EgMALl/V8AX6fommT1KAA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"5bb44a2e0d1933c8ef62b58c87fef06d4b52c2889253a1717819c66279b87b41","last_reissued_at":"2026-05-17T23:38:52.845449Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:52.845449Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Randomly replacing labels in in-context demonstrations barely hurts performance on classification and multiple-choice tasks across many models.","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Ari Holtzman, Hannaneh Hajishirzi, Luke Zettlemoyer, Mikel Artetxe, Mike Lewis, Sewon Min, Xinxi Lyu","submitted_at":"2022-02-25T17:25:19Z","abstract_excerpt":"Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choice tasks, consistently over 12 different models including GPT-3","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That randomly replacing labels does not introduce unintended statistical cues or that the chosen classification and multiple-choice tasks are representative of broader in-context learning behavior.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Randomly replacing labels in in-context demonstrations barely hurts performance, showing that label space, input distribution, and sequence format drive in-context learning more than ground-truth labels.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Randomly replacing labels in in-context demonstrations barely hurts performance on classification and multiple-choice tasks across many models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"61d4dfbf8396fe4eeb64d839bf014ddd2b62c460dba2dcabf580a436aeb1b12e"},"source":{"id":"2202.12837","kind":"arxiv","version":2},"verdict":{"id":"d0753e83-3c25-4144-8164-18aa9ee7932b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T09:47:47.546394Z","strongest_claim":"ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choice tasks, consistently over 12 different models including GPT-3","one_line_summary":"Randomly replacing labels in in-context demonstrations barely hurts performance, showing that label space, input distribution, and sequence format drive in-context learning more than ground-truth labels.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That randomly replacing labels does not introduce unintended statistical cues or that the chosen classification and multiple-choice tasks are representative of broader in-context learning behavior.","pith_extraction_headline":"Randomly replacing labels in in-context demonstrations barely hurts performance on classification and multiple-choice tasks across many models."},"references":{"count":237,"sample":[{"doi":"","year":2011,"title":"Robust Disambiguation of Named Entities in Text","work_id":"3474f56b-b2f4-4a79-969b-ce696ce45a7a","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2019,"title":"CODAH : An Adversarially-Authored Question Answering Dataset for Common Sense","work_id":"03c831cd-ef72-48f3-9e66-66368b214d6c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2005,"title":"Dolan, William B. and Brockett, Chris. Automatically Constructing a Corpus of Sentential Paraphrases. Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005). 2005","work_id":"30b6b875-c3df-419a-8022-45756fa18613","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia ,author=. Semantic Web ,year=","work_id":"742ea21d-e273-4d5b-af41-a056bd9f6844","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Abductive Commonsense Reasoning ,author=. ICLR ,year=","work_id":"696a87a7-3cc0-4345-b7e8-0a4048be94f4","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":237,"snapshot_sha256":"e0606b4aed025a4589039248ad69c666c1f7300df7f660b2b3fbf175f80bc8f5","internal_anchors":6},"formal_canon":{"evidence_count":2,"snapshot_sha256":"1a4bf141a32e3a1d3257f85429211d3c3a348e0ddeb32f97b209756120bf00e4"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2202.12837","created_at":"2026-05-17T23:38:52.845576+00:00"},{"alias_kind":"arxiv_version","alias_value":"2202.12837v2","created_at":"2026-05-17T23:38:52.845576+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2202.12837","created_at":"2026-05-17T23:38:52.845576+00:00"},{"alias_kind":"pith_short_12","alias_value":"LO2EULQNDEZ4","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"LO2EULQNDEZ4R33C","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"LO2EULQN","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":32,"internal_anchor_count":32,"sample":[{"citing_arxiv_id":"2604.16359","citing_title":"LLM4Log: A Systematic Review of Large Language Model-based Log Analysis","ref_index":118,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20457","citing_title":"The Structure and Dynamics of the Online MAHA-sphere","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17152","citing_title":"Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages","ref_index":143,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18930","citing_title":"OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2507.20906","citing_title":"Soft Head Selection for Injecting ICL-Derived Task Embeddings","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2401.03568","citing_title":"Agent AI: Surveying the Horizons of Multimodal Interaction","ref_index":123,"is_internal_anchor":true},{"citing_arxiv_id":"2509.24164","citing_title":"Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2511.01008","citing_title":"MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2511.13502","citing_title":"SnapAudit: Active Auditing of Differentially Private In-Context Learning via Snapshot-Based Simulation","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2512.13040","citing_title":"Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2210.03493","citing_title":"Automatic Chain of Thought Prompting in Large Language Models","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2302.01560","citing_title":"Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2501.09686","citing_title":"Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models","ref_index":98,"is_internal_anchor":true},{"citing_arxiv_id":"2604.16359","citing_title":"LLM4Log: A Systematic Review of Large Language Model-based Log Analysis","ref_index":118,"is_internal_anchor":true},{"citing_arxiv_id":"2305.03726","citing_title":"Otter: A Multi-Modal Model with In-Context Instruction Tuning","ref_index":62,"is_internal_anchor":true},{"citing_arxiv_id":"2604.16421","citing_title":"Measuring Representation Robustness in Large Language Models for Geometry","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12412","citing_title":"Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2211.12588","citing_title":"Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09678","citing_title":"Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2204.14198","citing_title":"Flamingo: a Visual Language Model for Few-Shot Learning","ref_index":77,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08904","citing_title":"OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces","ref_index":89,"is_internal_anchor":true},{"citing_arxiv_id":"2604.24334","citing_title":"Reducing Redundancy in Retrieval-Augmented Generation through Chunk Filtering","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2604.23371","citing_title":"When Context Sticks: Studying Interference in In-Context Learning","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.04243","citing_title":"Temporal Reasoning Is Not the Bottleneck: A Probabilistic Inconsistency Framework for Neuro-Symbolic QA","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2604.20146","citing_title":"SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition","ref_index":26,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV","json":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV.json","graph_json":"https://pith.science/api/pith-number/LO2EULQNDEZ4R33CWWGIP7XQNV/graph.json","events_json":"https://pith.science/api/pith-number/LO2EULQNDEZ4R33CWWGIP7XQNV/events.json","paper":"https://pith.science/paper/LO2EULQN"},"agent_actions":{"view_html":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV","download_json":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV.json","view_paper":"https://pith.science/paper/LO2EULQN","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2202.12837&json=true","fetch_graph":"https://pith.science/api/pith-number/LO2EULQNDEZ4R33CWWGIP7XQNV/graph.json","fetch_events":"https://pith.science/api/pith-number/LO2EULQNDEZ4R33CWWGIP7XQNV/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV/action/timestamp_anchor","attest_storage":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV/action/storage_attestation","attest_author":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV/action/author_attestation","sign_citation":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV/action/citation_signature","submit_replication":"https://pith.science/pith/LO2EULQNDEZ4R33CWWGIP7XQNV/action/replication_record"}},"created_at":"2026-05-17T23:38:52.845576+00:00","updated_at":"2026-05-17T23:38:52.845576+00:00"}