Pith. sign in

REVIEW 2 major objections 2 minor 25 cited by

Cross-Task Generalization via Natural Language Crowdsourcing Instructions

T0 review · 2 major / 2 minor · reviewed 2026-05-18 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models improve at new tasks by 19 percent when given human-written instructions.

desk verdict The paper builds a new meta-dataset of 61 tasks with their original instructions and shows a 19% gain on unseen tasks when instructions are provided, but the evidence for why that gain occurs is still thin. read the letter →

arxiv 2104.08773 v4 pith:QDJYZSFM submitted 2021-04-18 cs.CL cs.AIcs.CVcs.LG

Swaroop Mishra , Daniel Khashabi , Chitta Baral , Hannaneh Hajishirzi This is my paper
classification cs.CLcs.AIcs.CVcs.LG
keywords cross-taskgeneralizationnaturallanguageinstructionsinstructionfollowingmodelstounseentasksmetalearningcrowdsourcedNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper creates a collection of 61 different tasks drawn from existing NLP datasets, each paired with the original crowdsourced instructions that explain how to solve it. Models are trained on some of these tasks while seeing the instructions, then tested on the remaining tasks they have never encountered. The results show that including the instructions during training leads to better performance on the new tasks compared to training without them.

What carries the argument

A meta-dataset of 61 tasks with their human-authored instructions mapped to a unified schema, which is used to train models that encode the instruction text together with the input to produce the task output.

What would settle it

If replacing the actual instructions with irrelevant or shuffled text still produces the same performance gains on held-out tasks, the benefit would not be attributable to instruction understanding.

Watch

Extended reading notes

Core claim

Generative pre-trained language models that receive task instructions along with input data generate better outputs on tasks not seen during training, achieving a 19% improvement in generalization performance over models without access to instructions.

Load-bearing premise

The selected tasks are diverse and non-overlapping so that success on unseen ones reflects genuine understanding of the instructions rather than shared surface features or data overlap.

Editorial extensions

If this is right

  • Cross-task generalization can be directly measured by training on a subset of tasks and evaluating on completely held-out tasks.
  • Human-authored instructions provide a way to define tasks that transfers across different problems.
  • Models benefit from instructions specifically in the setting of unseen tasks rather than just seen ones.
  • Significant room remains for progress since current models fall short of an estimated upper bound performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Such models could eventually acquire capabilities in new domains by reading descriptions instead of needing large amounts of labeled examples for each.
  • This points toward training approaches that treat instructions as a general interface for task specification.
  • Extending the set of tasks beyond the current 61 could reveal whether the gains hold for more diverse or real-world problems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces NATURAL INSTRUCTIONS, a meta-dataset of 61 distinct NLP tasks with crowdsourced human-authored instructions and 193k input-output instances. Models are trained on a subset of seen tasks and evaluated on held-out unseen tasks; the central claim is that generative pre-trained language models that encode the task instructions along with the input achieve 19% relative improvement in generalization to unseen tasks compared to instruction-free baselines.

Significance. If the result holds after addressing the evaluation concerns, the work would demonstrate that natural language instructions can support cross-task generalization beyond what is possible with standard supervised learning on individual datasets. The release of the dataset and the unified schema for instructions would be a concrete resource for future research on instruction-following models. The empirical comparison of instruction-aware versus instruction-free conditions on explicitly held-out tasks is a strength when properly controlled.

major comments (2)
  1. [Evaluation] Evaluation setup: the claim that performance gains on held-out tasks can be attributed to instruction understanding rather than shared surface patterns requires evidence that the 61 tasks have no substantial overlap in category, output format, or instruction phrasing between train and test splits. No quantitative check (e.g., instruction embedding similarity or category-level leakage analysis) is reported, which is load-bearing for the central generalization claim.
  2. [Results] Results section: the abstract states a 19% relative gain but supplies no details on exact model sizes, training hyperparameters, number of runs, statistical significance tests, or task similarity controls. This leaves the magnitude and robustness of the reported improvement difficult to assess from the provided evidence.
minor comments (2)
  1. [Abstract] The estimated performance upperbound is mentioned in the abstract but not defined or computed in the main text; adding a brief description of how it is obtained would improve clarity.
  2. [Dataset] Notation for the unified schema of instructions could be introduced earlier with an example to help readers follow the dataset construction.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and describe the revisions we will make to improve the clarity and rigor of the evaluation and results sections.

read point-by-point responses

  1. Referee: [Evaluation] Evaluation setup: the claim that performance gains on held-out tasks can be attributed to instruction understanding rather than shared surface patterns requires evidence that the 61 tasks have no substantial overlap in category, output format, or instruction phrasing between train and test splits. No quantitative check (e.g., instruction embedding similarity or category-level leakage analysis) is reported, which is load-bearing for the central generalization claim.

    Authors: We agree that explicit evidence of limited overlap between seen and unseen tasks is important to support the attribution of gains to instruction understanding. The 61 tasks were selected from distinct existing NLP datasets with different objectives, output spaces, and instruction styles, which provides a natural separation. To strengthen this, we will add a quantitative analysis in the revised manuscript: we will compute average cosine similarities between instruction embeddings (using a sentence encoder) across train/test splits and include a category-level breakdown showing minimal leakage in task types or formats. This addition will directly address the concern. revision: yes

  2. Referee: [Results] Results section: the abstract states a 19% relative gain but supplies no details on exact model sizes, training hyperparameters, number of runs, statistical significance tests, or task similarity controls. This leaves the magnitude and robustness of the reported improvement difficult to assess from the provided evidence.

    Authors: We acknowledge that the abstract and results would benefit from more complete reporting. The manuscript already specifies the use of generative pre-trained models, but we will expand the results section and add an appendix detailing exact model sizes (T5 variants), training hyperparameters, number of runs with different random seeds, and statistical significance tests (e.g., paired t-tests across runs). The task similarity controls will be covered by the new analysis described in our response to the evaluation comment. These changes will make the reported 19% relative improvement easier to evaluate. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical cross-task evaluation is self-contained

full rationale

The paper constructs a new meta-dataset of 61 tasks with crowdsourced instructions and reports an empirical result: models trained on seen tasks generalize better to held-out unseen tasks when instructions are provided (19% improvement). This is measured directly via standard train/test splits on task instances, with no equations, fitted parameters, or derivations that reduce the reported gain to a quantity defined by the same data or by self-citation. The central claim rests on the experimental contrast between instruction-aware and instruction-free conditions on explicitly held-out tasks rather than any definitional equivalence or load-bearing self-reference.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the empirical performance difference being caused by instruction comprehension and on the assumption that the collected tasks form a representative sample of NLP problems with minimal unintended overlap.

assumptions (2)
  • domain assumption Pre-trained generative language models can meaningfully incorporate task instructions to improve output on held-out tasks.
    The modeling approach assumes current large language models possess sufficient capacity to use the provided instructions.
  • domain assumption The 61 tasks are sufficiently independent that training on a subset yields no direct knowledge of the held-out tasks.
    Required for the unseen-task evaluation to measure genuine generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Task Generalization via Natural Language Crowdsourcing Instructions." pith.science (2026). https://pith.science/paper/QDJYZSFM

@misc{pith2026210408773,
  author       = {Pith},
  title        = {Pith review of: Cross-Task Generalization via Natural Language Crowdsourcing Instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDJYZSFM}},
  note         = {Machine review of arXiv:2104.08773}
}
read the original abstract

Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

Discussion (0). Sign in to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    cs.CL 2022-02 accept novelty 8.0 of 10

    Randomly replacing labels in in-context demonstrations barely hurts performance, showing that label space, input distribution, and sequence format drive in-context learning more than ground-truth labels.

  2. PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data

    cs.CL 2025-12 conditional novelty 7.0 of 10

    PIAST iteratively optimizes few-shot examples in prompts via Monte Carlo Shapley value estimation, outperforming prior automatic prompting methods and setting new SOTA on classification, simplification, and GSM8K with...

  3. PRIMETIME : Limits of LLMs in Temporal Primitives

    cs.NE 2025-04 unverdicted novelty 7.0 of 10

    PRIMETIME generator reveals that LLM datetime parsing and arithmetic primitives are individually unreliable but fully learnable via fine-tuning, enabling frontier-level accuracy on event planning with small LoRA models.

  4. Self-Rewarding Language Models

    cs.CL 2024-01 conditional novelty 7.0 of 10

    Iterative self-rewarding via LLM-as-Judge in DPO training on Llama 2 70B improves instruction following and self-evaluation, outperforming GPT-4 on AlpacaEval 2.0.

  5. LIMA: Less Is More for Alignment

    cs.CL 2023-05 conditional novelty 7.0 of 10

    Fine-tuning a 65B model on 1,000 high-quality examples produces output that humans rate as good as or better than GPT-4 in 43% of cases, indicating most capabilities come from pretraining.

  6. VideoChat: Chat-Centric Video Understanding

    cs.CV 2023-05 conditional novelty 7.0 of 10

    VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.

  7. OPT: Open Pre-trained Transformer Language Models

    cs.CL 2022-05 unverdicted novelty 7.0 of 10

    OPT releases open decoder-only transformers up to 175B parameters that match GPT-3 performance at one-seventh the carbon cost, along with code and training logs.

  8. Multitask Prompted Training Enables Zero-Shot Task Generalization

    cs.LG 2021-10 conditional novelty 7.0 of 10

    Multitask fine-tuning of an encoder-decoder model on prompted datasets produces zero-shot generalization that often beats models up to 16 times larger on standard benchmarks.

  9. How Many Instructions Can LLMs Follow at Once?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    IFScale measures instruction-following at densities from 10 to 500 constraints and finds that even top frontier models satisfy only about two-thirds of 500 simultaneous keyword instructions.

  10. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    cs.CL 2024-06 conditional novelty 6.0 of 10

    MMLU-Pro is a revised benchmark that makes language model evaluation harder and more stable by using ten options per question and emphasizing reasoning over simple knowledge recall.

  11. Lessons from the Trenches on Reproducible Evaluation of Language Models

    cs.CL 2024-05 accept novelty 6.0 of 10

    The paper compiles practical lessons on reproducible LM evaluation and introduces the lm-eval library to mitigate common methodological problems in NLP.

  12. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    cs.SE 2024-03 unverdicted novelty 6.0 of 10

    LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.

  13. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

    cs.CL 2024-02 conditional novelty 6.0 of 10

    DPOP is a new loss function that prevents DPO from lowering preferred response likelihoods and outperforms standard DPO on diverse datasets, MT-Bench, and enables Smaug-72B to exceed 80% on the Open LLM Leaderboard.

  14. Gemini: A Family of Highly Capable Multimodal Models

    cs.CL 2023-12 conditional novelty 6.0 of 10

    Gemini Ultra reaches human-expert performance on MMLU for the first time and sets new state-of-the-art results on 30 of 32 benchmarks, including all 20 multimodal ones tested.

  15. Simple synthetic data reduces sycophancy in large language models

    cs.CL 2023-08 unverdicted novelty 6.0 of 10

    Scaling and instruction tuning increase sycophancy in LLMs on opinion and fact tasks, but a synthetic data fine-tuning intervention reduces it on held-out prompts.

  16. PandaGPT: One Model To Instruction-Follow Them All

    cs.CL 2023-05 conditional novelty 6.0 of 10

    A single model trained only on image-text pairs gains instruction-following ability across images, video, and audio by routing all modalities through ImageBind's shared embedding space into Vicuna.

  17. ART: Automatic multi-step reasoning and tool-use for large language models

    cs.CL 2023-03 unverdicted novelty 6.0 of 10

    ART automatically generates multi-step reasoning programs with tool integration for LLMs, yielding substantial gains over few-shot and auto-CoT prompting on BigBench and MMLU while matching hand-crafted CoT on most tasks.

  18. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

    cs.AI 2023-01 conditional novelty 6.0 of 10

    The Flan Collection demonstrates that task balancing, data enrichment, and mixed prompt training are critical to effective instruction tuning, yielding stronger Flan-T5 models released publicly.

  19. A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    A single LLM rewrite of skill descriptions using false positive and negative cases matches manual optimization performance in production, with most other pipeline components adding little value.

  20. Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    A multi-agent LLM system with majority voting achieves reported Micro F1 of 0.872 for delusion detection and 0.779 for classification on naturalistic speech transcripts.

  21. SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

    cs.CL 2025-02 unverdicted novelty 5.0 of 10

    SmolLM2 is a 1.7B-parameter language model that outperforms Qwen2.5-1.5B and Llama3.2-1B after overtraining on 11 trillion tokens using custom FineMath, Stack-Edu, and SmolTalk datasets in a multi-stage pipeline.

  22. Enhancing Memory Recall in LLMs with Gauss-Tin: A Hybrid Instructional and Gaussian Replay Approach

    cs.LG 2025-08 reject novelty 4.0 of 10

    Gauss-Tin, a replay method using a Gaussian mixture model with prompt-guided exemplar selection, reports positive backward transfer on the Natural Instructions benchmark versus sequential fine-tuning.

  23. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

  24. Large Language Models: A Survey

    cs.CL 2024-02 accept novelty 3.0 of 10

    The paper surveys key large language models, their training methods, datasets, evaluation benchmarks, and future research directions in the field.

  25. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Reference graph

Works this paper leans on

107 extracted references · 107 canonical work pages · cited by 25 Pith papers

  1. [1]

    Proceedings of EMNLP , pages=

    Learning from Task Descriptions , author=. Proceedings of EMNLP , pages=

    work page

  2. [3]

    Proceedings of NeurIPS , volume=

    SuperGLUE: A stickier benchmark for general-purpose language understanding systems , author=. Proceedings of NeurIPS , volume=

    work page

  3. [4]

    1960 , publisher=

    Programs with common sense , author=. 1960 , publisher=

    work page 1960

  4. [5]

    EMNLP , year=

    SQuAD: 100, 000+ Questions for Machine Comprehension of Text , author=. EMNLP , year=

    work page

  5. [7]

    Unsupervised Data Augmentation for Consistency Training , url =

    Xie, Qizhe and Dai, Zihang and Hovy, Eduard and Luong, Thang and Le, Quoc , booktitle =. Unsupervised Data Augmentation for Consistency Training , url =

    work page

  6. [9]

    AI Magazine , volume=

    From ‘F’to ‘A’on the NY Regents Science Exams: An Overview of the Aristo Project , author=. AI Magazine , volume=

    work page

  7. [10]

    Proceedings of NAACL-HLT , pages=

    How many data points is a prompt worth? , author=. Proceedings of NAACL-HLT , pages=

    work page

  8. [11]

    Logan IV and Eric Wallace and Sameer Singh , title =

    Taylor Shin and Yasaman Razeghi and Robert L. Logan IV and Eric Wallace and Sameer Singh , title =. Proceedings of EMNLP , pages =

    work page

Show all 107 references
  1. [12]

    arXiv preprint arXiv:2102.01335 , year=

    Neural Data Augmentation via Example Extrapolation , author=. arXiv preprint arXiv:2102.01335 , year=

  2. [13]

    arXiv preprint arXiv:1806.08730 , year=

    The natural language decathlon: Multitask learning as question answering , author=. arXiv preprint arXiv:1806.08730 , year=

    arXiv

  3. [14]

    2019 , url=

    The Natural Language Decathlon: Multitask Learning as Question Answering , author=. 2019 , url=

    2019

  4. [15]

    arXiv preprint arXiv:2004.05483 , year=

    Unsupervised commonsense question answering with self-talk , author=. arXiv preprint arXiv:2004.05483 , year=

    2004

  5. [16]

    arXiv preprint arXiv:2009.00751 , year=

    Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models , author=. arXiv preprint arXiv:2009.00751 , year=

    2009

  6. [17]

    Journal of instructional development , volume=

    The use of positive and negative examples during instruction , author=. Journal of instructional development , volume=. 1981 , publisher=

    1981

  7. [18]

    arXiv preprint arXiv:2103.10385 , year=

    GPT Understands, Too , author=. arXiv preprint arXiv:2103.10385 , year=

  8. [19]

    arXiv preprint arXiv:2001.07676 , year=

    Exploiting cloze questions for few-shot text classification and natural language inference , author=. arXiv preprint arXiv:2001.07676 , year=

    2001

  9. [20]

    arXiv preprint arXiv:2009.07118 , year=

    It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners , author=. arXiv preprint arXiv:2009.07118 , year=

    2009

  10. [21]

    arXiv preprint arXiv:1912.02164 , year=

    Plug and play language models: A simple approach to controlled text generation , author=. arXiv preprint arXiv:1912.02164 , year=

    1912

  11. [22]

    arXiv preprint arXiv:2103.11955 , year=

    Improving and Simplifying Pattern Exploiting Training , author=. arXiv preprint arXiv:2103.11955 , year=

  12. [23]

    arXiv preprint arXiv:2101.00190 , year=

    Prefix-Tuning: Optimizing Continuous Prompts for Generation , author=. arXiv preprint arXiv:2101.00190 , year=

    arXiv

  13. [24]

    Proceedings of the IEEE/CVF , pages=

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks , author=. Proceedings of the IEEE/CVF , pages=

  14. [25]

    arXiv preprint arXiv:2102.12206 , year=

    PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains , author=. arXiv preprint arXiv:2102.12206 , year=

  15. [26]

    Proceedings of NAACL , pages=

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences , author=. Proceedings of NAACL , pages=

  16. [27]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  17. [28]

    Proceedings of the 31st International Conference on Neural Information Processing Systems , pages=

    Prototypical networks for few-shot learning , author=. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages=

  18. [29]

    Proceedings of workshop on evaluation metrics and system comparison for automatic summarization , pages=

    An assessment of the accuracy of automatic evaluation in summarization , author=. Proceedings of workshop on evaluation metrics and system comparison for automatic summarization , pages=

  19. [30]

    Proceedings of the 2nd Workshop on Machine Reading for Question Answering , pages=

    Reasoning Over Paragraph Effects in Situations , author=. Proceedings of the 2nd Workshop on Machine Reading for Question Answering , pages=

  20. [32]

    Proceedings of ECCV , pages=

    Hard negative examples are hard, but useful , author=. Proceedings of ECCV , pages=. 2020 , organization=

    2020

  21. [33]

    2011 IEEE 11th International Conference on Data Mining , pages=

    Learning from negative examples in set-expansion , author=. 2011 IEEE 11th International Conference on Data Mining , pages=. 2011 , organization=

    2011

  22. [34]

    Proceedings of ICML Workshop on The Continuum from Labeled to Unlabeled Data , volume=

    Bootstrapped learning of semantic classes from positive and negative examples , author=. Proceedings of ICML Workshop on The Continuum from Labeled to Unlabeled Data , volume=

  23. [35]

    Proceedings of the VLDB Endowment , volume=

    Natural language to SQL: Where are we today? , author=. Proceedings of the VLDB Endowment , volume=. 2020 , publisher=

    2020

  24. [36]

    2013 , organization=

    Learning semantic maps from natural language descriptions , author=. 2013 , organization=

    2013

  25. [37]

    Proceedings of LREC , year=

    NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System , author=. Proceedings of LREC , year=

  26. [38]

    arXiv preprint arXiv:2101.06804 , year=

    What Makes Good In-Context Examples for GPT- 3 ? , author=. arXiv preprint arXiv:2101.06804 , year=

  27. [39]

    Machine learning , volume=

    Learning from natural instructions , author=. Machine learning , volume=. 2014 , publisher=

    2014

  28. [40]

    Proceedings of CoNLL , pages=

    Learning what is essential in questions , author=. Proceedings of CoNLL , pages=

  29. [41]

    Machine learning , volume=

    Multitask learning , author=. Machine learning , volume=. 1997 , publisher=

    1997

  30. [42]

    Proceedings of ACL , pages=

    CHARTDIALOGS: Plotting from Natural Language Instructions , author=. Proceedings of ACL , pages=

  31. [44]

    Proceedings of EMNLP: Findings , pages=

    Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections , author=. Proceedings of EMNLP: Findings , pages=

  32. [45]

    ArXiv , year=

    Towards General Purpose Vision Systems , author=. ArXiv , year=

  33. [46]

    Proceedings of IJCAI , year=

    Transformers as soft reasoners over language , author=. Proceedings of IJCAI , year=

  34. [47]

    Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Ves and Zettlemoyer, Luke , booktitle=

  35. [48]

    Khashabi, Daniel and Min, Sewon and Khot, Tushar and Sabharwal, Ashish and Tafjord, Oyvind and Clark, Peter and Hajishirzi, Hannaneh , booktitle=

  36. [49]

    Journal of Machine Learning Research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of Machine Learning Research , volume=

  37. [50]

    Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke S

    Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar S. Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke S. Zettlemoyer and Veselin Stoyanov , journal=. R o BERT a: A Robustly Optimized BERT Pretraining Approach. 2019 , url =

    2019

  38. [51]

    International Conference on Machine Learning , pages=

    Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  39. [52]

    Proceedings of NAACL-HLT , pages=

    Deep contextualized word representations , author=. Proceedings of NAACL-HLT , pages=

  40. [53]

    arXiv preprint arXiv:1906.04908 , year=

    Spoc: Search-based pseudocode to code , author=. arXiv preprint arXiv:1906.04908 , year=

    1906 arXiv

  41. [54]

    Proceedings of EMNLP-IJCNLP , pages=

    Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning , author=. Proceedings of EMNLP-IJCNLP , pages=

  42. [55]

    Going on a vacation

    “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding , author=. Proceedings of EMNLP-IJCNLP , pages=

  43. [56]

    Proceedings of EMNLP-IJCNLP , pages=

    Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning , author=. Proceedings of EMNLP-IJCNLP , pages=

  44. [57]

    Proceedings of the AAAI , year=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Proceedings of the AAAI , year=

  45. [58]

    Proceedings of NAACL , pages=

    DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs , author=. Proceedings of NAACL , pages=

  46. [59]

    Khashabi, Daniel and Stanovsky, Gabriel and Bragg, Jonathan and Lourie, Nicholas and Kasai, Jungo and Choi, Yejin and Smith, Noah A and Weld, Daniel S , journal=

  47. [60]

    Khot, Tushar and Clark, Peter and Guerquin, Michal and Jansen, Peter and Sabharwal, Ashish , booktitle=

  48. [61]

    A Call for Clarity in Reporting BLEU Scores

    Post, Matt. A Call for Clarity in Reporting BLEU Scores. Proceedings of the Third Conference on Machine Translation: Research Papers. 2018

    2018

  49. [62]

    Proceedings of ACL , pages=

    BLEURT: Learning Robust Metrics for Text Generation , author=. Proceedings of ACL , pages=

  50. [63]

    Text summarization branches out , pages=

    Rouge: A package for automatic evaluation of summaries , author=. Text summarization branches out , pages=

  51. [65]

    arXiv preprint arXiv:2010.12083 , year=

    Language-Conditioned Imitation Learning for Robot Manipulation Tasks , author=. arXiv preprint arXiv:2010.12083 , year=

    2010

  52. [67]

    arXiv preprint arXiv:2109.07830 , year=

    Reframing Instructional Prompts to GPTk's Language , author=. arXiv preprint arXiv:2109.07830 , year=

  53. [68]

    Proceedings of EMNLP , pages=

    Muppet: Massive Multi-task Representations with Pre-Finetuning , author=. Proceedings of EMNLP , pages=

  54. [69]

    Language Models are Few-Shot Learners , volume =

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom a...

  55. [70]

    Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pages=

    Prompt programming for large language models: Beyond the few-shot paradigm , author=. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pages=

    2021

  56. [71]

    Proceedings of ICLR , year=

    Multitask Prompted Training Enables Zero-Shot Task Generalization , author=. Proceedings of ICLR , year=

  57. [72]

    Proceedings of EMNLP , year=

    Few-Shot Text Generation with Natural Language Instructions , author=. Proceedings of EMNLP , year=

  58. [73]

    Proceedings of ICLR , year=

    Finetuned Language Models are Zero-Shot Learners , author=. Proceedings of ICLR , year=

  59. [74]

    Proceedings of EMNLP , year=

    CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP , author=. Proceedings of EMNLP , year=

  60. [75]

    Proceedings of ICML , pages=

    Calibrate before use: Improving few-shot performance of language models , author=. Proceedings of ICML , pages=

  61. [76]

    Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of EMNLP, pages 5799--5811

    2021

  62. [77]

    Ali M Ali. 1981. The use of positive and negative examples during instruction. Journal of instructional development, 5(1):2--7

    1981

  63. [78]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    2020 arXiv

  64. [79]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  65. [80]

    Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41--75

    1997

  66. [81]

    Pradeep Dasigi, Nelson F Liu, Ana Marasovic, Noah A Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of EMNLP-IJCNLP, pages 5927--5934

    2019

  67. [82]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of NAACL, pages 2368--2378

    2019

  68. [83]

    Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982

    2020

  69. [84]

    Kamath, Aniruddha Kembhavi, and Derek Hoiem

    Tanmay Gupta, A. Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2021. Towards general purpose vision systems. ArXiv, abs/2104.00743

    2021

  70. [85]

    Peter Hase and Mohit Bansal. 2021. When can models learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201

    2021

  71. [86]

    Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of EMNLP-IJCNLP, pages 2391--2401

    2019

  72. [87]

    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of NAACL, pages 252--262

    2018

  73. [88]

    Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what is essential in questions. In Proceedings of CoNLL, pages 80--89

    2017

  74. [89]

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: crossing format boundaries with a single qa system. In Proceedings of EMNLP: Findings, pages 1896--1907

    2020

  75. [90]

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composition . In Proceedings of AAAI

    2020

  76. [91]

    Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? In Proceedings of NAACL-HLT, pages 2627--2636

    2021

  77. [92]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL

    2019

  78. [93]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  79. [94]

    Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58--62

    2019

  80. [95]

    Winston Lin, Roman Yangarber, and Ralph Grishman. 2003. Bootstrapped learning of semantic classes from positive and negative examples. In Proceedings of ICML Workshop on The Continuum from Labeled to Unlabeled Data, volume 1, page 21

    2003

  81. [96]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586

    2021

  82. [97]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786

    2021

  83. [98]

    Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering

    2018

  84. [99]

    Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. 2020. Towards question format independent numerical reasoning: A set of prerequisite tasks. arXiv preprint arXiv:2005.08516

    2020

  85. [100]

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227--2237

    2018

  86. [101]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1--67

    2020

  87. [102]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1--7

    2021

  88. [103]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI

    2020

  89. [104]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  90. [105]

    Timo Schick and Hinrich Sch \"u tze. 2021. Few-shot text generation with natural language instructions. In Proceedings of EMNLP

    2021

  91. [106]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL, pages 7881--7892

    2020

  92. [107]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In Proceedings of ICLR

    2022

  93. [108]

    Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew Peters. 2020. Learning from task descriptions. In Proceedings of EMNLP, pages 1361--1375

    2020

  94. [109]

    Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. 2020. Hard negative examples are hard, but useful. In Proceedings of ECCV, pages 126--142. Springer

    2020

  95. [110]

    Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. In Proceedings of EMNLP

    2021

  96. [111]

    Qinyuan Ye and Xiang Ren. 2021. Zero-shot learning by generating task-specific adapters. arXiv preprint arXiv:2101.00420

    2021

  97. [112]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of ICML, pages 12697--12706

    2021

  98. [113]

    Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Proceedings of EMNLP: Findings, pages 2856--2878

    2021

  99. [114]

    going on a vacation

    Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of EMNLP-IJCNLP, pages 3354--3360

    2019

Pith tools

Reviewed May 18, 2026 · model on record in the stance chip above.