Pith. sign in

REVIEW 3 major objections 5 minor 47 references

LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that an encoder-only student model, DeBERTa-base-v3, improves from 28.9% to 39.3% average MMLU accuracy, a 10.4-point absolute gain, when fine-tuned on LLM-generated MCQA data whose answer choices are scored by the LLM.

desk verdict A useful empirical recipe for tiny MCQA models via LLM-generated data plus soft-label distillation, but the headline MMLU number is not yet protected against contamination or verified against a rigorous baseline. read the letter →

arxiv 2412.09807 v2 pith:JL3EWW22 submitted 2024-12-13 cs.CL

Patrick Sutanto , Joan Santoso , Esther Irawati Setiawan , Aji Prasetya Wibawa This is my paper · ORCID
classification cs.CL
keywords few-shotlearningmultiplechoicequestionansweringLLMdistillationsoftlabelsMMLUDeBERTasyntheticdatagenerationencoder-onlymodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a small encoder-only model can become an effective few-shot multiple-choice question answerer without needing large labeled datasets. It proposes a two-step recipe: use an LLM to generate synthetic MCQA questions and choices, and use the same LLM's probability scores over choices as soft labels when fine-tuning DeBERTa-v3-base. On MMLU this raises accuracy from 28.9% to 39.3%, a 10.4-point gain, and the 184M-parameter student outperforms much larger models reported under similar settings. The authors read this as evidence that LLM-driven data generation plus distillation transfers knowledge efficiently into small models.

What carries the argument

The load-bearing mechanism is the combination of LLM-based synthetic data generation and probability-score distillation. The teacher (Llama-3.1-8B-Instruct) creates MCQA instances either in JSON format or through a decomposed question/positive/negative generation pipeline, then outputs the likelihood of each answer letter, that is, a soft label over the choices. The student (DeBERTa-v3-base) is trained with a cross-entropy loss against these soft probabilities. The softness is what matters: with hard labels from the teacher the reported gain collapses, while with soft labels the student learns a smoother distribution, which the authors link to robustness against noisy generated data.

What would settle it

Evaluate the student on held-out MMLU subjects whose topics and styles were deliberately excluded from the 5-shot prompt and from the generation prompts; if accuracy on those subjects stays near the 28.9% baseline, the synthetic distribution did not transfer. A simpler test: replace the LLM's soft labels with random permutations of the same distribution and check whether the 10.4-point gain collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that knowledge distillation from LLMs works for few-shot MCQA in a specific, effective way: an LLM generates synthetic questions and choices, scores each choice with a soft probability, and those scores rather than hard labels are used to train a smaller encoder-only model. This recipe lifts DeBERTa-base-v3 from 28.9% to 39.3% on MMLU, surpassing the 5-shot performance of LLaMA-7B and Flan-T5-250M. The authors attribute the gain to the soft labels correcting noisy generated answers and effectively transferring the teacher's answer-choice distribution.

Load-bearing premise

The method assumes that the LLM-generated questions and choices, together with the LLM's probability scores, are a faithful proxy for the real MMLU test distribution, so that a student trained on this synthetic data learns genuine question-answering ability rather than overfitting to the synthetic data's quirks.

Editorial extensions

If this is right

  • A 184M-parameter student beats models over 30 times larger on few-shot MMLU, so deployment cost for MCQA systems in resource-constrained settings drops sharply.
  • The distillation step consistently improves accuracy across data sizes, generation methods, and teacher LLMs, indicating that soft-label training stabilizes learning from imperfect synthetic data.
  • With roughly 1,024 generated items, the student matches the performance of training on 256 to 512 real ARC examples, reducing the need for expensive labeled MCQA data.
  • The JSON generation variant, despite lower parse success, yields slightly better accuracy and combines well with an existing multi-task DeBERTa baseline, suggesting an implicit quality filter.
  • The approach transfers across datasets (MMLU to ARC) and extends to binary classification via a logit-threshold heuristic, pointing toward broader applicability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the synthetic distribution faithfully matches real test data, the recipe could generalize to any task where an LLM can propose candidate outputs and score them, with the student's ceiling bounded by the teacher's domain coverage.
  • Editorial extension: the large gain from soft labels over hard labels implies that other synthetic-data pipelines with noisy generation should treat teacher confidence as part of the training signal, a design choice the paper validates only for MCQA and ARC.
  • Editorial extension: the cross-dataset experiment suggests a substantial portion of the improvement is format learning plus broad language understanding; a sharper test would vary topic overlap while holding format constant, which the paper does not perform.
  • Editorial extension: because generation and scoring still require a multi-GPU LLM, the computational savings materialize at inference time, not training time, so the practical benefit is mainly for deployment on memory-constrained devices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a few-shot MCQA pipeline in which an instruction-tuned LLM (Llama-3.1-8B-Instruct) generates synthetic multiple-choice questions from five seed examples, either directly in JSON or through a decomposed question/positive/negative generation scheme; the same LLM then scores the choices by extracting token logits; and the soft scores are used as distillation targets when fine-tuning DeBERTa-v3-base. The central reported result is on MMLU: JSON generation plus distillation raises DeBERTa's accuracy from 28.9% to 39.3%, a 10.4-point absolute gain, with ablations on ARC-Easy/ARC-Challenge covering generated-data size, generation temperature, generator model size, and a paraphrasing baseline.

Significance. If the headline result is robust, the paper offers a practical and inexpensive recipe for building an efficient encoder-only MCQA system from only five labeled examples. The work has real strengths: it ablates the number of generated points, generation temperature, and number of negative choices; it compares against a paraphrasing baseline; it includes a cross-dataset transfer experiment (Appendix C.8); and it reports results averaged over five seeds in the ablation tables. However, the two load-bearing issues below — the absence of an MMLU contamination analysis and the absence of variance on the headline MMLU table — prevent me from endorsing the central claim as stated. Both are empirically addressable, so the paper warrants major revision rather than rejection.

major comments (3)
  1. [Section 4.1 / Appendix C.9] The headline result in Section 4.1 (the 28.9 to 39.3 MMLU gain) is not protected against test-set contamination. Appendix C.9 reports semantic similarity between generated, training, and test questions only for ARC-Easy and ARC-Challenge, not for the MMLU tasks that carry the main claim. Because the teacher model, Llama-3.1-8B-Instruct, is trained on web-scale data and almost certainly has seen MMLU test items, and because Appendix B.2 shows that several MMLU tasks produce fewer than 1024 usable points (e.g., college mathematics 512, formal logic 538, high school US history 305), the reported 10.4-point improvement could in part reflect the student memorizing test-like content through the generated questions and teacher scores. The authors should run the same similarity analysis per MMLU task, or provide an equivalent contamination check, and report the numbers before the MMLU result can be taken at face value.
  2. [Section 4, Table 1] The paper says 'We average the results across 5 different random seeds for all experiments,' but Table 1 gives no standard deviation, standard error, or per-seed values for any MMLU row. The ARC ablations in Table 4 show that the 5-shot baseline has very high seed variance (26.5 ± 13.8 on ARC-Easy), so without error bars on the MMLU numbers the 10.4-point gain over the 5-shot baseline cannot be distinguished from seed noise. Please add variance or significance measures for Table 1 and for the MMLU rows of Table 10.
  3. [Section 4.1, Table 1] The comparison with LLaMA-7B (35.1) and Flan-T5-250M (35.9) is confounded: those numbers are taken from the original papers, which use different prompting, few-shot selection, and possibly different evaluation splits, as the table footnote admits. The claims in the abstract and Section 4.1 that the distilled DeBERTa 'surpasses' or 'approaches' these models are therefore not established by a controlled experiment. The authors should either run the same evaluation protocol on LLaMA-7B and Flan-T5-250M (or an open checkpoint with the same harness) or rephrase those claims to say 'comparable to published numbers.'
minor comments (5)
  1. [Abstract and Section 1] The phrase 'trained with only a 5-shot examples' is ungrammatical; it should be 'trained with only five examples' or 'with only 5-shot examples.' The same typo appears in Section 4.1.
  2. [Appendix C.6] There is a typo: 'We observer that' should be 'We observe that.'
  3. [Appendix B.2] The function name 'vmlDeviceGetMemoryInfo' appears to be a typo for 'nvmlDeviceGetMemoryInfo' (or a similar pyNVML call).
  4. [Section 3.2 and Appendix C.2] The main-text distillation loss in Section 3.2 uses no temperature parameter, but Appendix C.2 introduces a temperature r and reports results for different values. The paper should state explicitly which distillation temperature was used for the MMLU experiments in Section 4.
  5. [Reproducibility] The manuscript does not include a link to code or the generated datasets, which limits reproducibility of the exact reported numbers. I would encourage the authors to release both.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the distillation targets are teacher outputs on generated data, and the MMLU accuracy is an external held-out measurement, not an input.

full rationale

The derivation chain is self-contained. The headline result (MMLU accuracy rising from 28.9% to 39.3%) is an externally measured accuracy of DeBERTa-v3-base after training on LLM-generated MCQA examples with LLM soft-label distillation. The soft targets are defined by the paper's own equations (Ldistill = LCE(p, p̂), with p = softmax(ŷ^LLM_c) derived from the teacher's token likelihoods), and the student is trained on those targets; no quantity is fitted to the MMLU test set and then renamed a prediction. The 5-shot dev set is the only MMLU input used for data generation, and the test split is held out. There are no load-bearing self-citations: the distillation and scoring references (Hinton et al.; Robinson & Wingate) are external standard methods, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The potential concern that Llama-3.1-8B-Instruct may have memorized MMLU content during pretraining, together with the fact that the Appendix C.9 similarity check is run only on ARC rather than MMLU, is an external-validity/contamination threat, not a circularity: it does not make the reported accuracy equivalent to an input by construction. Under the stated rubric, that concern belongs in correctness risk and does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on standard ML hyperparameters (generation temperature, data count, negative count, distillation temperature) chosen via ablations, and on domain assumptions about the transferability of LLM-generated data and the teacher's logits. No new theoretical entities are introduced.

free parameters (4)
  • Generation temperature = 2.0
    Chosen based on ARC ablations; higher temperature increases data diversity but lowers JSON parse success.
  • Number of generated data points = 1024
    Default; ablations show accuracy improves with more data, though JSON method sometimes falls short on long tasks (e.g., 305 for high school US history).
  • Number of negative examples (decomposed) = 5
    Ablations show robustness to this choice; 5 is used in main MMLU experiments.
  • Distillation temperature = 1
    Softmax temperature for soft labels; ablations show non-zero temperatures outperform hard labels (temperature 0), and values 0.5 to 2 are similar.
assumptions (4)
  • domain assumption The synthetic MCQA data generated by Llama-3.1-8B and its probability scores are a faithful proxy for the real MMLU test distribution.
    The method's improvements on MMLU depend on the student learning from generated data that resembles the test set; Section 4.1 and Appendix C.9 provide indirect evidence via semantic similarity, but do not prove distributional match.
  • domain assumption DeBERTa-v3-base, when trained on question-choice pairs, is agnostic to choice order, so the decompose method's always-first-correct label is harmless.
    The decomposed generation always places the correct answer as the first choice; the authors assert the student does not exploit position because each choice is scored independently (Appendix F). If this is false, the decompose training signal would be invalid.
  • domain assumption The teacher LLM's logits for the choice characters are valid soft-label probabilities for the MCQ answers.
    The scoring method directly follows Robinson & Wingate (2023); it assumes the logits reflect meaningful answer likelihoods and not arbitrary calibration artifacts.
  • domain assumption The teacher LLM did not memorize MMLU test items in a way that leaks through generated data.
    The semantic similarity analysis (Appendix C.9) shows generated questions are not near-duplicates of test questions, but this does not rule out paraphrased memorization. If test answers leaked through the teacher's soft labels, the significance of the result would be reduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering." pith.science (2026). https://pith.science/paper/JL3EWW22

@misc{pith2026241209807,
  author       = {Pith},
  title        = {Pith review of: LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JL3EWW22}},
  note         = {Machine review of arXiv:2412.09807}
}
read the original abstract

Multiple Choice Question Answering (MCQA) is an important problem with numerous real-world applications, such as medicine, law, and education. The high cost of building MCQA datasets makes few-shot learning pivotal in this domain. While Large Language Models (LLMs) can enable few-shot learning, their direct application in real-world scenarios is often hindered by their high computational cost. To address this challenge, we propose a simple yet effective approach that uses LLMs for data generation and scoring. Our approach utilizes LLMs to create MCQA data which contains questions and choices, and to assign probability scores to the generated choices. We then use the generated data and LLM-assigned scores to finetune a smaller and more efficient encoder-only model, DeBERTa-v3-base by leveraging distillation loss. Extensive experiments on the Massive Multitask Language Understanding (MMLU) benchmark demonstrate that our method improves accuracy from 28.9% to 39.3%, representing a gain of over 10% compared to a baseline finetuned directly on 5-shot examples. This shows the effectiveness of LLM-driven data generation and knowledge distillation for few-shot MCQA.

Figures

Figures reproduced from arXiv: 2412.09807 by the authors.

Figure 1
Figure 1. Framework for Few-Shot MCQA using LLM-Generated Data and Distillation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Effect of Generated Data Size on Few-Shot MCQA Accuracy. The figure compares the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Average Maximum Cosine Similarity between Generated Questions and the Training/Test [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Maximum Cosine Similarity Observed between Generated Questions and the Train [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 12 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024

    2024

  3. [3]

    Gemini: A family of highly capable multimodal models

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1, 2023 a

    arXiv 2023

  4. [4]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023 b

    arXiv 2023

  5. [5]

    Generating questions and multiple-choice answers using semantic analysis of texts

    Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. Generating questions and multiple-choice answers using semantic analysis of texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp.\ 1125--1136, 2016

    work page 2016

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165, 2020

    arXiv 2005

  7. [7]

    Disco: Distilling counterfactuals with large language models

    Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. Disco: Distilling counterfactuals with large language models. arXiv preprint arXiv:2212.10534, 2022

    arXiv 2022

  8. [8]

    Chatgpt versus human in generating medical graduate exam multiple choice questions—a multinational prospective study (hong kong sar, singapore, ireland, and the united kingdom)

    Billy Ho Hung Cheung, Gary Kui Kai Lau, Gordon Tin Chun Wong, Elaine Yuen Phin Lee, Dhananjay Kulkarni, Choon Sheong Seow, Ruby Wong, and Michael Tiong-Hong Co. Chatgpt versus human in generating medical graduate exam multiple choice questions—a multinational prospective study (hong kong sar, singapore, ireland, and the united kingdom). PloS one, 18 0 (8)...

    work page 2023

Show all 47 references
  1. [9]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  2. [10]

    Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140, 2023

    2023 arXiv

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    A survey of data augmentation approaches for nlp

    Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. arXiv preprint arXiv:2105.03075, 2021

    2021 arXiv

  7. [15]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  9. [17]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  10. [18]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11 0 (14): 0 6421, 2021

    2021

  11. [19]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947, 2016

    2016 arXiv

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    Chatgpt prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review

    Yavuz Selim K yak and Emre Emekli. Chatgpt prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgraduate medical journal, pp.\ qgae065, 2024

    2024

  14. [22]

    Datasets: A community library for natural language processing

    Quentin Lhoest, Albert Villanova Del Moral, Yacine Jernite, Abhishek Thakur, Patrick Von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. arXiv preprint arXiv:2109.02846, 2021

    2021 arXiv

  15. [23]

    Self-alignment with instruction backtranslation

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2023

    2023 arXiv

  16. [24]

    Distractor generation for multiple choice questions using learning to rank

    Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel, and C Lee Giles. Distractor generation for multiple choice questions using learning to rank. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pp.\ 284--290, 2018

    2018

  17. [25]

    D2llm: Decomposed and distilled large language models for semantic search

    Zihan Liao, Hang Yu, Jianguo Li, Jun Wang, and Wei Zhang. D2llm: Decomposed and distilled large language models for semantic search. arXiv preprint arXiv:2406.17262, 2024

    2024 arXiv

  18. [26]

    Roberta: A robustly optimized bert pretraining approach

    Y Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  19. [27]

    On llms-driven synthetic data generation, curation, and evaluation: A survey

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126, 2024

    2024 arXiv

  20. [28]

    Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp.\ 6448--6458

    Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp.\ 6448--6458. PMLR, 2020

    2020

  21. [29]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  22. [30]

    Leveraging large language models for multiple choice question answering

    Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=yKbprarjc5B

    2023

  23. [31]

    End-to-end generation of multiple-choice questions using text-to-text transfer transformer models

    Ricardo Rodriguez-Torrealba, Eva Garcia-Lopez, and Antonio Garcia-Cabot. End-to-end generation of multiple-choice questions using text-to-text transfer transformer models. Expert Systems with Applications, 208: 0 118258, 2022

    2022

  24. [32]

    tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework

    Damien Sileo. tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference o...

    2024

  25. [33]

    Automatic generation of multiple choice questions using wikipedia

    Arjun Singh Bhatia, Manas Kirti, and Sujan Kumar Saha. Automatic generation of multiple choice questions using wikipedia. In Pattern Recognition and Machine Intelligence: 5th International Conference, PReMI 2013, Kolkata, India, December 10-14, 2013. Proceedings 5, pp.\ 733--7...

    2013

  26. [34]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2818--2826, 2016

    2016

  27. [35]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  28. [36]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  29. [37]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  30. [38]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  31. [39]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods i...

    2020

  32. [40]

    A survey on knowledge distillation of large language models

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024

    2024 arXiv

  33. [41]

    Genie: Achieving human parity in content-grounded datasets generation

    Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. Genie: Achieving human parity in content-grounded datasets generation. arXiv preprint arXiv:2401.14367, 2024

    2024 arXiv

  34. [42]

    Enhancing distractor generation for multiple-choice questions with retrieval augmented pretraining and knowledge graph integration

    Han-Cheng Yu, Yu-An Shih, Kin-Man Law, Kai-Yu Hsieh, Yu-Chen Cheng, Hsin-Chih Ho, Zih-An Lin, Wen-Chuan Hsu, and Yao-Chung Fan. Enhancing distractor generation for multiple-choice questions with retrieval augmented pretraining and knowledge graph integration. arXiv preprint ar...

    2024 arXiv

  35. [43]

    When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings

    Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial inte...

    2021

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [47]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.