Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Basic Reading Distillation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small model trained to imitate a large model's reading drills rivals models 15 to 26 times its size.

desk verdict Useful distillation recipe that improves a small model, but the headline '26x larger teacher' comparison rests on a table mixing 73-task and 5-task BIG-bench averages. read the letter →

arxiv 2507.19741 v2 pith:T7KAGQWC submitted 2025-07-26 cs.CL

Zhi Zhou , Sirui Miao , Xiangyu Duan , Hao Yang , Min Zhang This is my paper · ORCID
classification cs.CL
keywords basicreadingdistillationknowledgetasksmalllanguagemodelsnamedentityrecognitionquestionansweringCC-100corpustransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes basic reading distillation (BRD), a step that sits before both knowledge distillation and task distillation: a teacher LLM is prompted to read each sentence of a generic corpus (here, five million CommonCrawl sentences) by naming entities and by raising and answering questions about content, structure, or attitude, and a small student model is trained to imitate those behaviors. The claim is that this 'reading education' transfers comprehension ability, not task skill, so the student performs better on task-agnostic benchmarks than the same model trained further on raw text, and approaches or beats models 15 to 26 times larger, including the teacher itself in some settings. The paper further shows BRD is orthogonal to existing distillation techniques: adding it to knowledge distillation, task distillation, or supervised fine-tuning improves each. The central comparison is a 564M-parameter XGLM student educated by a 13B Vicuna teacher, evaluated on NLI, paraphrase, boolean QA, sentiment, and 73 BIG-bench tasks.

What carries the argument

The carrier of the argument is the reading-annotation passage itself. For each sentence $s_i$, the teacher is prompted to output a named-entity recognition result $\mathrm{NER}(s_i)$ and a question-answer pair $\mathrm{QRA}(s_i)$, and the student's training data interleaves these with the original sentence in passage form (e.g., $s_1 <\text{sep}> \mathrm{NER}(s_1) <\text{sep}> s_2 <\text{sep}> \mathrm{NER}(s_2) \ldots$), alongside the unannotated original passages to stabilize pretrained weights. The QRA prompt restricts questions to content, structure, or attitude. Cross-entropy comparisons show the student's distribution moves closer to the teacher's after BRD, and layer-wise probes show the improvement is spread across layers, which the paper takes as evidence that the education changes the model rather than merely copying surface outputs.

What would settle it

Construct a held-out benchmark whose answer vocabulary is disjoint from any term the teacher could generate in NER or content questions (e.g., arithmetic or direction tasks), and measure whether XGLM-BRD still beats continued next-token training on the same corpus. If the gain vanishes whenever no annotation happens to overlap with task-relevant vocabulary, the general-reading explanation fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a small model educated to imitate an LLM's basic reading behaviors on ordinary, task-unrelated sentences acquires generalized task ability far beyond what continued next-token training on the same sentences provides. Concretely, the 564M-parameter XGLM-BRD, trained on teacher-generated named-entity and question-answer annotations for five million CC-100 sentences, averages 54.2% on six evaluation groups in the blind setting versus 48.0% for the untrained student and 51.8% for MiniLLM, and matches or exceeds the 15x larger XGLM-7.5B; in the relaxed setting, applying BRD again on downstream task inputs gives XGLM-BRD2 an average of 67.1%, close to Vicuna-13B's 67.6%. The paper interprets this as convincing evidence that explicit, interpretable reading behaviors are a distillation medium that existing logit/implicit or task-imitation approaches leave unused.

Load-bearing premise

The load-bearing premise is that the teacher-generated reading annotations on generic sentences are genuinely unrelated to the downstream tasks, so any accuracy gain is attributable to general reading ability rather than leaked task-relevant information; Appendix A.6 shows that removing sentiment-related questions and answers from the QRA data drops SST-2 accuracy by four points, which directly strains that premise.

Editorial extensions

If this is right

  • A 564M-parameter student, after BRD, reaches an average of 54.2% across six task groups in the blind setting, beating the 15x larger XGLM-7.5B and both KD baselines, suggesting task-agnostic distillation can substitute for model scale on many benchmarks.
  • Adding BRD to task distillation improves the average from 62.0% to 67.1%, and adding it to SFT improves from 80.3% to 81.8%, so BRD behaves as an orthogonal component on top of both mainstream distillation paradigms.
  • Because every sentence can seed an annotation, BRD removes the data-scale and diversity ceiling that limits task distillation, enabling distillation from practically unlimited web-mined text.
  • The cross-entropy decrease between student and teacher distributions and the layer-wise probe improvements imply the benefit is structural, not a shallow imitation of the teacher's output strings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'task-unrelated' framing is stronger than the evidence supports: the sentiment ablation shows part of the SST-2 gain comes from sentiment-labeled QA annotations, so the method likely transfers attention towards task-relevant information as well as general comprehension.
  • The teacher-generation step is itself costly (a 13B model annotating five million sentences); a testable extension would compare BRD against directly training the student on distilled task data of equivalent compute, or against cheaper synthetic annotations from a smaller teacher.
  • One could map which reading behaviors matter by ablating NER versus QRA per task family; the paper's ablation only covers the relaxed setting and QRA appears more important, but a task-by-task decomposition might reveal where NER carries the load.
  • If the effect is distributional, applying BRD in the student's own language mix or on code and structured data (with analogous reading behaviors like identifier-to-description Q&A) may extend the recipe beyond natural language benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes basic reading distillation (BRD), a two-stage method in which a teacher LLM (Vicuna-13B) is prompted to produce named entity recognition and question-raising/answering annotations for individual sentences from CC-100, and a small student (XGLM-564M) is autoregressively trained on passages that interleave original sentences, NER annotations, and QRA annotations. The student is then evaluated on XNLI, RTE, CB, PAWS-X, BoolQ, SST-2, and 73 BIG-bench multiple-choice tasks in three settings: blind test, relaxed test (task training-set inputs available without gold labels), and with downstream supervision. The paper reports that the resulting XGLM-BRD2 approaches or surpasses Vicuna-13B on average, that BRD improves over further training on raw text, that it can be combined with knowledge distillation and task distillation, and that it shifts the student's distribution toward the teacher, as measured by cross entropy.

Significance. The paper addresses a real problem: making distillation data scalable and task-agnostic. Its central empirical finding, if supported, would be practically valuable: a 564M model educated on general reading behaviors can compete with a 13B teacher on several benchmarks. The paper is unusually thorough in its ablations: it tests a second teacher (Llama3.1-8B), removes sentiment-related annotations (A.6), ablates NER vs. QRA (Table 8), compares passage- vs. sentence-level training (Table 7), and includes layer-wise probing (A.5). These analyses are a genuine strength. However, the headline claims rest on aggregate numbers that mix BIG-bench task sets of different sizes, and the 'unrelated data' claim is partly contradicted by the paper's own A.6 results. The core idea is credible and the manuscript is close, but the evaluation needs to be reworked before the claims are supported.

major comments (3)
  1. [§4.3, Table 3] The Average column in Table 3 is computed over seven columns, one of which (BIG-bench-Avg) is not comparable across rows because the daggered rows use a reduced 5-task BIG-bench set while Vicuna-13B and XGLM-7.5B use the full 73-task set. As a concrete check, the six non-BIG-bench columns alone give XGLM-BRD2 an average of 70.9 and Vicuna-13B an average of 72.9, so the reported aggregate closeness (67.1 vs. 67.6) is driven by the unmatched BIG-bench entries. The claims in Section 4.3 and the conclusion that the student is 'comparable or superior to the 26x bigger teacher' should be recomputed on a matched task set, or the BIG-bench columns should be reported separately for the full and reduced sets.
  2. [§3.1 and Appendix A.6] The prompt in Table 2 explicitly asks questions about the 'content, structure, or attitude of the sentence', and A.6 shows that removing questions/answers about attitude lowers SST-2 from 78.1 to 74.2. Consequently, the BRD education data are not 'unrelated' to the downstream sentiment task as claimed in the abstract and Section 1; the blind-test improvement on SST-2 is partly attributable to task-relevant annotations. The authors should either exclude attitude questions entirely from QRA generation, or quantify the leakage effect for each downstream task and soften the 'unrelated data' claim accordingly.
  3. [§4.1 and §4.3] The blind-test comparison between XGLM-BRD and the KD baselines (SKD and MiniLLM) is not controlled: the released KD students are GPT-2 760M models, while XGLM-BRD is initialized from XGLM-564M. The two base models differ in architecture, pretraining data, and tokenizer, so the observed differences cannot be attributed to the distillation method. Table 4 reports GPT-2 + BRD combinations, which partially addresses this, but the main results should include an XGLM-564M-based KD baseline or clearly present Table 4 as the controlled comparison.
minor comments (5)
  1. [Tables 3-8] The column heading 'PA WS-X' contains an extra space in multiple tables; it should read 'PAWS-X'.
  2. [§4.3] The model name appears as 'XGLM-BRD 2' in two places; the rest of the paper uses 'XGLM-BRD2', so the notation should be unified.
  3. [§5.2, Table 5] The cross-entropy computation is described only informally; please specify how the token-level probabilities are aligned and averaged for sequences of different lengths, and report the number of texts N.
  4. [Figure 2] Please specify whether the x-axis is the number of original sentences or the number of annotated passages, and report the variance across seeds or training runs to support the claimed plateau.
  5. [Abstract and §6] The text says 'over 20x bigger' in the abstract and conclusion, '26x bigger' in Section 4.3, and '15x bigger' for XGLM-7.5B; it would help to make these size comparisons consistent and state the parameter counts being compared.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: BRD's reported gains are empirical outcomes, not quantities defined in terms of the reported results.

full rationale

The paper's pipeline is empirical: teacher-generated NER/QRA passages on CC-100 sentences are used as training data, and downstream accuracy is then measured directly. No parameter is fitted to the downstream tasks and then reported as a prediction; the cross-entropy analysis in Section 5.2 is a post-hoc diagnostic rather than the source of the claim. The orthogonality claim in Section 5.1 is supported by combination experiments, not by definition, and no uniqueness theorem or load-bearing self-citation is invoked. The main non-circularity caveats are correctness/leakage issues: Appendix A.6 admits that attitude questions in the QRA data are 'related to the SST-2 task' and that removing them drops SST-2 by 4 points, which weakens the 'unrelated to downstream tasks' framing; and Table 3's daggered BIG-bench averages mix a 5-task reduced set with the teacher's 73-task average, making the aggregate '26x bigger teacher' comparison not apples-to-apples. Neither issue makes a result equivalent to its own input by construction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method relies on several unexamined choices: the teacher model, the annotation prompts, the data size, and the mix ratio. The most consequential is the inclusion of attitude questions in QRA, which introduces task-specific signal.

free parameters (5)
  • training_sentence_count = 5 million
    Number of CC-100 sentences used for BRD data synthesis; chosen by authors, not swept or justified.
  • teacher_model = Vicuna-13B
    Choice of teacher for generating annotations.
  • student_model = XGLM-564M
    Choice of student architecture.
  • mixed_data_ratio = unspecified
    The ratio of original passages to BRD passages is not given.
  • training_hyperparameters = lr=3e-4, batch=8, max_len=2048, max_steps=40000
    Hyperparameters chosen without sensitivity analysis; final checkpoint selection not described.
assumptions (3)
  • domain assumption Teacher-generated NER and QRA labels are accurate enough to teach the student
    The method assumes the Vicuna-13B annotations are high-quality supervision; no quality check is reported.
  • domain assumption Training on generic text reading behaviors transfers to downstream tasks
    The central hypothesis; the paper tests it empirically but assumes no task-specific information is needed.
  • domain assumption Cross entropy between teacher and student distributions is a valid proxy for ability transfer
    Used in Sec 5.2 to support the mechanism; not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Basic Reading Distillation." pith.science (2026). https://pith.science/paper/T7KAGQWC

@misc{pith2026250719741,
  author       = {Pith},
  title        = {Pith review of: Basic Reading Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7KAGQWC}},
  note         = {Machine review of arXiv:2507.19741}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable abilities in various natural language processing areas, but they demand high computation resources which limits their deployment in real-world. Distillation is one technique to solve this problem through either knowledge distillation or task distillation. Both distillation approaches train small models to imitate specific features of LLMs, but they all neglect basic reading education for small models on generic texts that are \emph{unrelated} to downstream tasks. In this paper, we propose basic reading distillation (BRD) which educates a small model to imitate LLMs basic reading behaviors, such as named entity recognition, question raising and answering, on each sentence. After such basic education, we apply the small model on various tasks including language inference benchmarks and BIG-bench tasks. It shows that the small model can outperform or perform comparable to over 20x bigger LLMs. Analysis reveals that BRD effectively influences the probability distribution of the small model, and has orthogonality to either knowledge distillation or task distillation.

Figures

Figures reproduced from arXiv: 2507.19741 by the authors.

Figure 1
Figure 1. The illustration of BRD process. act as the teacher models. Specifically, knowledge distillation (Hinton et al., 2015) usually trains the student models to imitate implicit features inside the teacher models, while task distillation (Chen et al., 2020) usually trains the student models to imitate explicit behaviors of the teacher models. Different to both distillation approaches, we pro￾pose basic reading distillati… view at source ↗
Figure 2
Figure 2. The performance curve along with different [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The accuracy comparison between BRD and the baseline on BIG-bench tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The results of probing XGLM-564M and XGLM-BRD layer-wisely on the downstream tasks in the blind [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages

  1. [1]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. https://arxiv.org/abs/2306.13649 On-policy distillation of language models: Learning from self-generated mistakes . Preprint, arXiv:2306.13649

    arXiv 2024

  2. [2]

    Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2023. https://arxiv.org/abs/2211.08264 Qameleon: Multilingual qa with only 5 examples . Preprint, arXiv:2211.08264

    arXiv 2023

  3. [3]

    Yonatan Belinkov. 2022. https://doi.org/10.1162/coli_a_00422 Probing classifiers: Promises, shortcomings, and advances . Computational Linguistics, 48(1):207--219

    doi:10.1162/coli_a_00422 2022

  4. [4]

    BIG bench authors. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research

    2023

  5. [5]

    Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. https://api.semanticscholar.org/CorpusID:11253972 Model compression . In Knowledge Discovery and Data Mining

    work page 2006

  6. [6]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. https://arxiv.org/abs/2307.03109 A survey on evaluation of large language models . Preprint, arXiv:2307.03109

    arXiv 2023

  7. [7]

    Shijie Chen, Yu Zhang, and Qiang Yang. 2024. https://doi.org/10.1145/3663363 Multi-task learning in natural language processing: An overview . ACM Comput. Surv., 56(12)

    work page doi:10.1145/3663363 2024

  8. [8]

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. 2020. Big self-supervised models are strong semi-supervised learners. In Advances in neural information processing systems, 33:22243–22255

    work page 2020

Show all 41 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  2. [10]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  3. [11]

    Bowman, Holger Schwenk, and Veselin Stoyanov

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Asso...

    2018

  4. [12]

    Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. https://api.semanticscholar.org/CorpusID:203595067 The commitmentbank: Investigating projection in naturally occurring discourse . In Proceedings of Sinnund Bedeutung 23

    2019

  5. [13]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023 a . https://api.semanticscholar.org/CorpusID:259164722 Knowledge distillation of large language models . ArXiv, abs/2306.08543

    2023 arXiv

  6. [14]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.267 Pre-training to learn in context . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4849--4870, Toron...

    2023 doi

  7. [15]

    Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. https://arxiv.org/abs/2305.15717 The false promise of imitating proprietary llms . Preprint, arXiv:2305.15717

    2023 arXiv

  8. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  9. [17]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. https://doi.org/10.18653/v1/2023.acl-long.830 Large language models are reasoning teachers . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852--14882,...

    2023 doi

  10. [18]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. https://doi.org/10.18653/v1/2023.findings-acl.507 Distilling step-by-step! outperforming larger language models with less training d...

    2023 doi

  11. [19]

    Jie Huang and Kevin Chen-Chuan Chang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.67 Towards reasoning in large language models: A survey . In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049--1065, Toronto, Canada. Association for Comput...

    2023 doi

  12. [20]

    Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. 2022. Weighted distillation with unlabeled examples. In Advances in neural information processing systems

    2022

  13. [21]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.372 T iny BERT : Distilling BERT for natural language understanding . In Findings of the Association for Computational Linguist...

    2020 doi

  14. [22]

    Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.242 BERT - EMD : Many-to-many layer mapping for BERT compression with earth mover ' s distance . In Proceedings of the 2020 Conference on Empirica...

    2020 doi

  15. [23]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  16. [24]

    Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. https://doi.org/10.18653/v1/2020.acl-main.537 F ast BERT : a self-distilling BERT with adaptive inference time . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguist...

    2020 doi

  17. [25]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. https://doi.org/10.18653/v1/2023.acl-short.151 Teaching small language models to reason . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  18. [26]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. https://arxiv.org/abs/2306.02707 Orca: Progressive learning from complex explanation traces of gpt-4 . Preprint, arXiv:2306.02707

    2023 arXiv

  19. [27]

    Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2019. Knockoff nets: Stealing functionality of black-box models. In CVPR

    2019

  20. [28]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Emp...

    2013

  21. [29]

    Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Shu Zhao, Peng Zhang, and Jie Tang. 2023. https://doi.org/10.18653/v1/2023.acl-industry.15 GKD : A general knowledge distillation framework for large-scale pre-trained language model . In Proceedings of the 61st Annual Me...

    2023 doi

  22. [30]

    Eric Wallace, Mitchell Stern, and Dawn Song. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.446 Imitation attacks and defenses for black-box machine translation systems . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), page...

    2020 doi

  23. [31]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  24. [32]

    Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2023. Pinto: Faithful language reasoning using prompt-generated rationales. In Eleventh International Conference on Learning Representations

    2023

  25. [33]

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021 a . https://doi.org/10.18653/v1/2021.findings-emnlp.354 Want to reduce labeling cost? GPT -3 can help . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195--4205, Punta ...

    2021 doi

  26. [34]

    Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021 b . https://doi.org/10.18653/v1/2021.findings-acl.188 M ini LM v2: Multi-head self-attention relation distillation for compressing pretrained transformers . In Findings of the Association for Computational Lin...

    2021 doi

  27. [35]

    Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023. https://arxiv.org/abs/2301.11796 Call for papers -- the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus . Preprint, arXiv:2301.11796

    2023 arXiv

  28. [36]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. https://doi.org/10.18653/v1/N19-1131 PAWS : Paraphrase adversaries from word scrambling . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Tec...

    2019 doi

  29. [37]

    Zheyu Zhang, Han Yang, Bolei Ma, David R \"u gamer, and Ercong Nie. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.13 Baby`s C o T hought: Leveraging large language models for enhanced reasoning in compact models . In Proceedings of the BabyLM Challenge at the 27th Confer...

    2023 doi

  30. [38]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2023. https://arxiv.org/...

    2023 arXiv

  31. [39]

    Qinhong Zhou, Zonghan Yang, Peng Li, and Yang Liu. 2023. https://doi.org/10.18653/v1/2023.acl-long.738 Bridging the gap between decision and logits in decision-based knowledge distillation for pre-trained language models . In Proceedings of the 61st Annual Meeting of the Assoc...

    2023 doi

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.