Pith. sign in

REVIEW 4 major objections 5 minor 32 references

When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that when training data is tiny, student models that teach each other can do as well as, or better than, a student taught by a large pretrained teacher, and that the simplest teacher-less variant scores highest on syntax.

desk verdict A credible but under-powered BabyLM submission: the teacher-less vs teacher-guided comparison is real but statistically unbacked, and the paper's most defensible finding is that simple self-distillation wins. read the letter →

arxiv 2411.16487 v1 pith:3DKNSXGB submitted 2024-11-25 cs.CL cs.AI

Srikrishna Iyer This is my paper · ORCID
classification cs.CLcs.AI
keywords BabyLMknowledgedistillationmutuallearningself-distillationbi-leveloptimizationdata-efficientpretrainingRoBERTasyntaxbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether, on the tiny text corpora of the BabyLM challenge, a group of student models trained to teach each other can do as well as a student trained by a large pretrained teacher. It claims they can. Its proposed method, Diversity Induced Weighted Mutual Learning (DWML), trains several compact RoBERTa students of different sizes, with a learned weight for each student, and improves syntactic scores over RoBERTa-base by about 2-3.4 percentage points on the 10M and 100M tracks. The paper also reports that a much simpler teacher-less method, self-distillation, achieves the best syntactic numbers among all distillation variants. This matters because removing the teacher removes the largest memory and compute cost of distillation, which is attractive when deployment targets are resource-constrained edge systems.

What carries the argument

Diversity Induced Weighted Mutual Learning (DWML) is the central mechanism. It replaces the single teacher-student pipeline with p peer students whose parameter targets are N/(i+1) fractions of the teacher's size; Bayesian optimization searches for concrete RoBERTa architectures (layers, heads, hidden size) near those targets. Training alternates an inner loop that optimizes all student parameters with a loss combining cross-entropy on labels and KL divergence toward peers, and an outer loop that optimizes student importance weights on the probability simplex using mirror descent. That dual loop is what lets the model decide which peers to trust, and the paper reports R = 0.7 between learned weights and student accuracy. The same machinery is ablated to test the number of peers, the label-versus-peer balance α, and the value of dynamic weights.

What would settle it

Train the same DWML, self-distillation, and teacher-guided KD pipelines directly on the official 2024 BabyLM 10M and 100M corpora with several random seeds; if teacher-guided KD or the challenge baselines match or beat the teacher-less methods there, or if seed variance exceeds the reported effect sizes, the paper's central conclusion about small data would not transfer.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a teacher is not needed for small-data language model pretraining. A set of peer students with different architectures, trained jointly so each student's prediction is pulled toward an ensemble weighted by learned importance, matches or beats distillation from a fixed RoBERTa-base teacher on syntax benchmarks. Concretely, DWML raises BLiMP from 49.6% to 51.6% on the 10M track and from 49.8% to 52.1% on the 100M track, and BLiMP Supplement from 48.9% to 52.3% on 10M and from 46.8% to 48.4% on 100M. In the method comparison on 10M data, DWML scores 51.58% on BLiMP Filtered versus 47.65% for teacher-guided KD, while plain self-distillation scores highest at 51.73% on BLiMP Filtered and 56.53% on BLiMP Supplement. The paper therefore concludes that teacher-less methods can match or exceed teacher-supervised approaches, though not necessarily through complex peer-learning machinery.

Load-bearing premise

The paper assumes that the 2023 BabyLM corpus is similar enough to the 2024 challenge corpus that models trained on 2023 data and scored by the 2024 pipeline speak to the 2024 challenge, even though the corpora are only described as "appear similar" after removing QED and increasing CHILDES.

Editorial extensions

If this is right

  • If correct, teacher-less pretraining can replace teacher-guided distillation on small corpora without sacrificing syntactic accuracy, removing the cost of training and storing a large teacher.
  • The learned peer weights could be used as an architectural search signal: models that receive higher weights are also the more accurate ones.
  • The fact that plain self-distillation beats the more elaborate DWML on syntax suggests that, on small data, the main benefit may come from regularization internal to one model rather than from multi-model agreement.
  • The efficiency picture is mixed: DWML cuts GPU utilization by roughly 32% relative to RoBERTa-base but takes longer to train, so the practical win depends on whether memory or wall-clock time is the bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the models were trained on the 2023 BabyLM corpus and scored with the 2024 challenge pipeline, a direct rerun on the 2024 corpus is the cleanest test of whether these conclusions transfer.
  • Beyond the paper: the per-task tables show large swings across peers, so the headline gaps may be within seed or peer variance; repeated runs with several random seeds would settle that.
  • Beyond the paper: the bi-level weighting scheme is not tied to language modeling, so the same design could be tested on small-data image classification or speech tasks where teacher cost is also prohibitive.
  • Beyond the paper: a minimal extension would compare DWML against a single student of identical total parameter budget trained alone, separating the benefit of peer sharing from the benefit of having an architecture search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Diversity Induced Weighted Mutual Learning (DWML) for small-corpus language model pretraining. Student architectures are found by Bayesian optimization to match target parameter counts N/(i+1), and peers are trained jointly with a weighted loss combining cross-entropy and KL divergence; peer weights are updated by mirror descent using a bi-level hypergradient. The authors train RoBERTa-base-sized and smaller models on the 2023 BabyLM 10M and 100M corpora and evaluate with BLiMP, BLiMP Supplemental, EWoK, and GLUE. They report that teacher-less methods (SD and DWML) match or exceed teacher-guided KD/KD_DWML on syntactic benchmarks, though their proposed DWML is second to the simpler SD, and they conclude that peer-learning complexity is not necessary.

Significance. If the empirical claims hold, the paper provides evidence relevant to sample-efficient pretraining: a teacher-less, dynamically weighted mutual-learning scheme can rival a fixed larger teacher at lower GPU utilization and FLOPs. The manuscript is honest about SD beating DWML, and it ships model and code links and uses the standard BabyLM evaluation pipeline. The main contributions—diverse student search and bi-level weighting—are interesting, but their validation is currently incomplete due to missing statistical support and an under-derived hypergradient; the significance is therefore conditional on revision.

major comments (4)
  1. [§5.1, Table 2] The headline comparison in Table 2 is not statistically supported. Section 5.1 states that results are "mean statistics over multiple runs," but no number of seeds, standard deviations, confidence intervals, or per-run scores are reported anywhere, including the detailed Appendix D tables. The decisive margins are small relative to the noise apparent in the table itself: e.g., SD vs KD on BLiMP Filtered is 51.73 vs 47.65, DWML vs KD_DWML is 51.58 vs 47.47, and several sub-scores in Table 6 hover near chance and vary by more than 10 points across peers. Moreover, the "Best" column in Table 2 is the maximum over four peer architectures, so each method's reported score is an oracle over architectures selected on the evaluation benchmark. A run-to-run standard deviation of 1–3 points would be enough to erase or reverse the claimed teacher-less advantage. The authors should report per-seed results with a fixed architecture-selection rule (e.g., a held-out development split) and provide confidence intervals before the central claim can be evaluated.
  2. [§3.2, Eq. (3)] The hypergradient formula in Eq. (3) is asserted without derivation and contains an undefined coefficient γ; the text says "as shown in the paper" without identifying the source. As written the expression is also dimensionally unclear: ∂L2/∂ω_i is a scalar, while (∂L2/∂θ)(∂La/∂θ)^T is an outer-product matrix if θ is a vector, so the equality cannot hold without further context (e.g., an implicit trace/inner product or a scalarization via the chain rule through the inner-loop optimization). In addition, Eq. (4) appears to evaluate the gradient at the updated weight ω^{k+1}_i, which is circular if taken literally. Because the outer-loop weight update is the core novelty of DWML, the authors need to derive Eq. (3) from the bi-level problem, define La and L2 precisely, specify γ and the mirror-descent step size η, and correct the index in Eq. (4).
  3. [§4.2] The models are pretrained on the 2023 BabyLM corpus, not on the 2024 challenge corpus used for the stated submission, and the authors justify this by saying the corpora "appear similar" after removing QED and increasing CHILDES. No quantitative distribution-shift analysis is provided. If the 2024 corpus differs in domain mix or noise, the comparison to the challenge baselines in Table 1 and the conclusions about small-data distillation may not transfer to the actual challenge setting. The authors should either validate the proxy (e.g., compare corpus statistics or train a small probe on both corpora) or clearly reframe the paper as a 2023-corpus study.
  4. [§5.3, Fig. 2(b)] The hyperparameter α=0.5 is selected by comparing α=0.3, 0.5, and 0.7 on the same BLiMP and BLiMP Supplemental benchmarks that are later used for the main results in Table 2. This is tuning on the target evaluation metric, and the reported DWML scores are therefore optimistic relative to a fully pre-registered or held-out selection procedure. The authors should use a development split for hyperparameter selection or report the selection effect; the same concern applies to the "Best" column in Table 2, which selects peer architectures on the evaluation benchmark.
minor comments (5)
  1. [§2, References] The citation "Zhang et al." appears multiple times without a year, and the corresponding reference list entry lacks a year and venue; please complete the citation.
  2. [Algorithm 1, lines 3–5] θ∗_i is defined as the solution of an architecture-size minimization, but line 5 says "Initialize peer model i with parameters θ∗_i"; this conflates architecture hyperparameters with network weights and should be reworded to "initialize with the architecture found by the search."
  3. [Algorithm 1, line 4] The constraint "layers | heads" is inconsistent with the text, which requires the embedding dimension to be divisible by the number of attention heads; the constraint should read "heads | dim."
  4. [Table 3] The entries "Training steps 150", "Mini batch size 3", and "Num. of mini batches 60" do not unambiguously define the schedule; please clarify whether 150 is the number of epochs or optimizer steps and how the 100M-word corpus is covered by these numbers.
  5. [§5.3, Fig. 2(c)] The reported correlation R=0.7 is based on only four points; please report the associated p-value or confidence interval, or refrain from calling the relationship "near-perfect."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported teacher-less advantage is a held-out empirical result, not an input-equivalent construction.

full rationale

The paper's central comparisons are evaluated on external BabyLM benchmarks (BLiMP, BLiMP Supplement, EWoK, GLUE) using models pretrained on fixed corpora. The method-defining equations (Eq. 1 for target student sizes, Eq. 2 for the loss, Eq. 3 for the hypergradient, and Eq. 4 for the mirror-descent weight update) do not contain the reported benchmark scores as inputs, so the claim that teacher-less methods can match or surpass teacher-supervised approaches is an evaluated outcome rather than a tautology. The weighted-mutual-learning loss and hypergradient are explicitly attributed to external prior work by Zhang et al., not to the present author, and no load-bearing argument rests on a self-citation. Two methodological weaknesses noted in the paper are not circularity: the alpha=0.5 hyperparameter was selected using the same evaluation benchmarks in Section 5.3, and the 'Best' column in Table 2 is the maximum over four peer architectures without error bars. These affect statistical reliability, but they do not make the reported scores equivalent to a fitted parameter or to a self-referential definition. Similarly, the Section 4.2 assumption that the 2023 BabyLM corpus is similar to the 2024 corpus is an external-validity concern rather than a circular derivation. I find no step in which a predicted quantity is defined in terms of itself or forced by construction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method introduces no new theoretical entities. Its novelty is limited to using Bayesian optimization to pick diverse student architectures; the loss, hypergradient, and mirror descent are inherited from Zhang et al. The central comparison depends on several hand-set hyperparameters (alpha, eta, gamma, peer count) that are not fixed by external evidence.

free parameters (5)
  • alpha (peer vs label loss weight) = 0.5
    Selected from ablation on {0.3, 0.5, 0.7} using the same BLiMP and BLiMP Supplement benchmarks as the final evaluation (Section 5.3), rather than a held-out validation set.
  • eta (mirror descent step size) = not reported
    Used in Eq. 4 to update student importance weights; the paper only says 'step size with annealing' and provides no value or schedule.
  • gamma (hypergradient coefficient) = undefined
    Appears in the hypergradient formula Eq. 3 but is never defined in the paper; needed to compute the outer-loop weight gradient.
  • number of peer students M = 4 for 10M, 2 for 100M
    Chosen by hand; Table 3 marks the 4-peer model as selected for 10M and the 2-peer model for 100M. Ablations show 4 peers give slightly better BLiMP Supplement but 2 peers are more efficient.
  • student target sizes N/(i+1) = 60M, 42M, 34M, 28M
    Parameter-count targets set by Eq. 1; architectures matching these sizes are found by Bayesian optimization over layers, heads, and hidden size.
assumptions (3)
  • domain assumption Hypergradient formula Eq. 3 correctly computes the outer-loop gradient for the importance weights
    Stated as 'As shown in the paper' with no derivation; copied from Zhang et al. and depends on an undefined constant gamma.
  • domain assumption BabyLM 2023 corpus is an adequate proxy for the 2024 BabyLM challenge corpus
    Section 4.2: 'it appears to be similar to the dataset released for the 2024 challenge', with QED removed and more CHILDES, but no measured check of distribution shift.
  • domain assumption 150 training steps (with batch size 3 and 60 mini-batches per step) are sufficient to reach meaningful comparison
    The training schedule in Table 3 is ambiguous and very short; no convergence checks are provided, yet all methods are compared under this schedule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?." pith.science (2026). https://pith.science/paper/3DKNSXGB

@misc{pith2026241116487,
  author       = {Pith},
  title        = {Pith review of: When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DKNSXGB}},
  note         = {Machine review of arXiv:2411.16487}
}
read the original abstract

We present our submission to the BabyLM challenge, aiming to push the boundaries of data-efficient language model pretraining. Our method builds upon deep mutual learning, introducing a student model search for diverse initialization. We address the limitation of treating students equally by formulating weighted mutual learning as a bi-level optimization problem. The inner loop learns compact students through online distillation, while the outer loop optimizes weights for better knowledge distillation from diverse students. This dynamic weighting strategy eliminates the need for a teacher model, reducing computational requirements. Our evaluations show that teacher-less methods can match or surpass teacher-supervised approaches.

Figures

Figures reproduced from arXiv: 2411.16487 by the authors.

Figure 1
Figure 1. Overview of the difference between Vanilla knowledge distillation and our approach, Diversity induced [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison across different experimental settings for 10M dataset: (left) varying number of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Peer importance weights dynamically trained using mirror descent algorithm as described in Equation [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: GPU utilization for different distillation methods. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 15 canonical work pages

  1. [1]

    Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. http://www.lrec-conf.org/proceedings/lrec2014/pdf/877_Paper.pdf The AMARA corpus: Building parallel language resources for the educational domain . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC '14) , pages 1856--1862, Reykjavik, Ic...

    work page 2014

  2. [2]

    Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. 2020. https://doi.org/10.1609/aaai.v34i04.5746 Online Knowledge Distillation with Diverse Peers . Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3430--3437

    doi:10.1609/aaai.v34i04.5746 2020

  3. [3]

    [ Call for Papers ] The 2nd BabyLM Challenge : Sample -efficient pretraining on a developmentally plausible corpus

    Leshem Choshen, Ryan Cotterell, Michael Y Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. [ Call for Papers ] The 2nd BabyLM Challenge : Sample -efficient pretraining on a developmentally plausible corpus

    work page

  4. [4]

    Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. BinaryConnect : training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS '15, pages 3123--3131, Cambridge, MA, USA. MIT Press

    work page 2015

  5. [5]

    Mathias Creutz. 2018. Open subtitles paraphrase corpus for six languages. arXiv preprint arXiv:1809.06142

    work page Pith review arXiv 2018

  6. [6]

    Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang. 2020. https://proceedings.neurips.cc/paper/2020/hash/91c77393975889bd08f301c9e13a44b7-Abstract.html Agree to Disagree : Adaptive Ensemble Knowledge Distillation in Gradient Space . In Advances in Neural Information Processing Systems , volume 33, pages 12345--12355....

    work page 2020

  7. [7]

    Martin Gerlach and Francesc Font-Clos. 2020. A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics. Entropy, 22(1):126

    2020

  8. [8]

    Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 , NIPS '15, pages 1135--1143, Cambridge, MA, USA. MIT Press

    work page 2015

Show all 32 references
  1. [9]

    Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representations. arXiv preprint arXiv:1511.02301

    2015 arXiv

  2. [10]

    Hinton, O

    Geoffrey E. Hinton, O. Vinyals, and J. Dean. 2015. https://www.semanticscholar.org/paper/Distilling-the-Knowledge-in-a-Neural-Network-Hinton-Vinyals/0c908739fbff75f03469d13d4a1a07de3414ee19 Distilling the Knowledge in a Neural Network . ArXiv

    2015

  3. [11]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2024

  4. [12]

    Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H

    Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyürek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshu...

    2024 doi arXiv

  5. [13]

    Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. 2018. https://proceedings.neurips.cc/paper_files/paper/2018/hash/f33ba15effa5c10e873bf3842afb46a6-Abstract.html Neural Architecture Search with Bayesian Optimisation and Optimal Transpo...

    2018

  6. [14]

    Xiaojie Li, Jianlong Wu, Hongyu Fang, Yue Liao, Fei Wang, and Chen Qian. 2020. https://doi.org/10.1007/978-3-030-58610-2_2 Local Correlation Consistency for Knowledge Distillation . In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ...

    2020 doi

  7. [15]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://doi.org/10.48550/arXiv.1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach . arXiv preprint. ArXiv:1907.11692 [cs]

    2019 doi arXiv

  8. [16]

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. https://doi.org/10.1109/ICCV.2017.298 Learning Efficient Convolutional Networks through Network Slimming . In 2017 IEEE International Conference on Computer Vision ( ICCV ) , pages 2755-...

    2017 doi

  9. [17]

    Brian MacWhinney. 2000. The CHILDES project: Tools for analyzing talk: Transcription format and programs, Vol . 1, 3rd ed . The CHILDES project: Tools for analyzing talk: Transcription format and programs, Vol . 1, 3rd ed. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US...

    2000

  10. [18]

    Fernando Nogueira. 2014--. https://github.com/bayesian-optimization/BayesianOptimization Bayesian Optimization : Open source constrained global optimization tool for Python

    2014

  11. [19]

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. 2021. https://doi.org/10.1145/3447582 A Comprehensive Survey of Neural Architecture Search : Challenges and Solutions . ACM Comput. Surv., 54(4):76:1--76:34

    2021 doi

  12. [20]

    David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.146 Trained on 100 million words and still in shape: BERT meets British National Corpus . In Findings of the Association for Computational Linguistics : EACL 202...

    2023 doi

  13. [21]

    Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. https://aclanthology.org/J00-3003 Dialogue act modeling for automatic tagging and recognition of conversatio...

    2000

  14. [22]

    Inar Timiryasov and Jean-Loup Tastet. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.24 Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty . In Proceedings of the BabyLM Challenge at the 27th Conference o...

    2023 doi

  15. [23]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. https://proceedings.neurips.cc/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html SuperGLUE : A Stickier Benchmark for General - Purpose L...

    2019

  16. [24]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A Multi - Task Benchmark and Analysis Platform for Natural Language Understanding . In Proceedings of the 2018 EMNLP Workshop BlackboxNLP : A...

    2018 doi

  17. [25]

    Luting Wang, Xiaojie Li, Yue Liao, Zeren Jiang, Jianlong Wu, Fei Wang, Chen Qian, and Si Liu. 2022. https://arxiv.org/abs/2207.05345v1 HEAD : HEtero - Assists Distillation for Heterogeneous Object Detectors

    2022 arXiv

  18. [26]

    Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.1 Findings of the BabyLM Challenge : Sample - Efficie...

    2023 doi

  19. [27]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl_a_00321 BLiMP : The Benchmark of Linguistic Minimal Pairs for English . Transactions of the Association for Computational Linguistics, ...

    2020 doi

  20. [28]

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019. https://arxiv.org/abs/1905.08094 Be your own teacher: Improve the performance of convolutional neural networks via self distillation . Preprint, arXiv:1905.08094

    2019 arXiv

  21. [29]

    Weighted Mutual Learning with Diversity - Driven Model Compression

    Miao Zhang, Li Wang, David Campos, Wei Huang, Chenjuan Guo, and Bin Yang. Weighted Mutual Learning with Diversity - Driven Model Compression

  22. [30]

    Hospedales, and Huchuan Lu

    Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2018. https://doi.org/10.1109/CVPR.2018.00454 Deep Mutual Learning . In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 4320--4328, Salt Lake City, UT. IEEE

    2018

  23. [31]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.