DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization

Hexuan Deng; Jing Li; Min Zhang; Wenxiang Jiao; Xuebo Liu; Zhaopeng Tu

REVIEW 4 major objections 5 minor 13 references

DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DRPruning claims that dynamically reweighting training data by a scaling-law-predicted reference loss restores balanced performance across domains after structured pruning, outperforming static scheduling in perplexity and downstream tasks.

desk verdict A solid, well-controlled data-scheduling method for pruning recovery, but the headline overclaims against official Sheared Llama at 2.7B and the reference-loss prediction needs direct validation. read the letter →

arxiv 2411.14055 v2 pith:KTG4F5DK submitted 2024-11-21 cs.CL

Hexuan Deng , Wenxiang Jiao , Xuebo Liu , Jing Li , Min Zhang , Zhaopeng Tu This is my paper · ORCID

classification cs.CL

keywords structuredpruningdistributionallyrobustoptimizationdataschedulingscalinglawscontinuedpretrainingmultilingualrobustnessperplexityLLMefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DRPruning is a data-scheduling method for structured pruning and continued pretraining of large language models that targets the uneven performance drop pruning causes across domains. The paper argues that the standard Sheared Llama-style schedule, which forces the pruned model to track the loss profile of a larger model, does not constrain the space of possible data distributions and leaves some domains behind. DRPruning instead runs distributionally robust optimization with two automatic adjustments: a scaling-law fit predicts each domain's loss at the end of training and uses that as the reference loss, and the reference data ratio itself drifts toward domains whose losses deviate most. In monolingual and multilingual experiments, the method reports lower perplexity, higher downstream task scores, and better instruction-tuning win rates than reproduced baselines, with almost no added training cost. The point is that a pruned model's recovery can be steered by its own predicted loss landscape rather than by hand-set DRO hyperparameters.

What carries the argument

The carrying object is the DRO update loop that couples three quantities per domain: the current validation loss, a predicted reference loss, and a reference data ratio. The reference loss comes from a four-parameter scaling-law fit, refit after each evaluation once 20% of training is done, and the reference ratio is updated after 40% of training by an exponential moving average of the current DRO weights, constrained by a chi-square divergence ball around the initial ratio. That loop is what lets the method reallocate data toward underperforming domains without collapsing into training only on the highest-loss domain.

What would settle it

Compare the predicted reference loss from the scaling-law fit at the 20% checkpoint with the actual end-of-training loss per domain; if the predicted minimum for slow-converging domains is systematically too low or too high, the DRO weight updates are miscalibrated and the perplexity gain over fixed scheduling should disappear when the extrapolation is replaced by the true final loss.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that dynamic DRO data scheduling, with a scaling-law-predicted reference loss and a gradually updated reference data ratio, restores balanced performance across domains after structured pruning. During training, evaluation losses per domain refit the curve $\hat{\ell}(P,T)=A\cdot P^{-\alpha}T^{-\beta}+E$, and the fitted curve's value at the final training step is taken as the minimum acceptable performance for that domain; the DRO update then raises weight for domains whose current loss exceeds this reference. At the same time, the reference data ratio is moved toward the current DRO weights via $p^{t+1}_R=\delta q^t+(1-\delta)p^t_R$, inside a chi-square ball constraint, with each domain's reference ratio bounded between $1/n$ and $n$ times its initial value. The paper reports average gains of $-5.59\%$ in perplexity, $+1.52\%$ in downstream tasks, and a $55.4\%$ instruction-tuning win rate, with the largest downstream gain ($+2.95\%$) under multilingual distribution shift, plus a $+17.9\%$ improvement on a domain-level sentence continuation benchmark it constructs.

Load-bearing premise

The load-bearing premise is that the four-parameter scaling-law fit extrapolates each domain's final loss reliably from partial curves taken after 20% of training, and that the reproduced Sheared Llama baseline is a faithful stand-in for the original; if either fails, the DRO weights and the head-to-head comparison are miscalibrated.

Editorial extensions

If this is right

Pruned models trained with DRPruning converge faster during the pruning phase and reach lower perplexity than constant or Sheared Llama-style schedules, with the gap widening in later training.
The same dynamic schedule transfers from monolingual pruning to multilingual continued pretraining, where it yields a +2.95% average downstream gain over the reproduced Sheared Llama baseline.
DRPruning introduces no extra GPU compute; data-ratio updates run on CPU and add less than 1.5% to total training time, and parallelizing them does not change performance.
Domain-level sentence continuation built from unlabeled data shows the method improves underperforming domains while keeping other domains stable, so perplexity gains are not achieved by sacrificing balance.
Fine-grained domain segmentation, tested by splitting CC and C4 into smaller clusters, further accelerates early convergence and improves final cross-entropy, suggesting the method benefits from finer-grained supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's head-to-head evidence is against ReSheared, its own reproduction of Sheared Llama, not against the officially released Sheared Llama 2.7B model; the appendix states the 2.7B reproduction was not possible under the available data, so the superiority claim over the original 2.7B model rests on the reproduction's fidelity.
If the scaling-law reference loss is as reliable as reported, the same recipe could replace hand-set DRO hyperparameters in other imbalanced training settings, including pretraining from scratch and cross-domain post-training, which the paper lists as untested future directions.
The instruction-tuning machine translation experiment suggests the dynamic data schedule transfers beyond language modeling; a direct test would be applying it to RLHF-style preference data, where domain balance is known to matter.
The mask-similarity analysis shows pruning decisions vary substantially across seeds with no significant perplexity difference, so part of the measured benefit may come from the continued-pretraining data schedule rather than from which substructures get pruned; the paper does not fully separate those effects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The method behind DRPruning is genuinely useful: applying group DRO to pruning recovery, with the reference loss set by a scaling-law extrapolation and the reference data ratio updated gradually, is a sensible new combination. The paper does the controlled comparison the right way — reproducing Sheared Llama on the same data (ReSheared) as the primary baseline, adding a multilingual setting, instruction tuning, a domain-level benchmark, and reporting that the overhead is under 1.5% of training time. Code is promised, and the authors are explicit that they could not reproduce the official 2.7B Sheared Llama.

The soft spots are in the claims, not the engineering. The abstract says DRPruning surpasses similarly sized models, but at 2.7B the official Sheared Llama still beats it on the 15-task average (52.53 vs 51.63) and on the authors' own domain benchmark (89.32 vs 86.00). The controlled win over ReSheared is real and consistent, but the headline needs to be narrowed to what the evidence supports.

The deeper methodological worry is the reference loss. Eq. 4 is a four-parameter fit to the same loss curve the DRO loop is shaping. Calling it a prediction of the end-of-training loss is generous; it is an extrapolation that has not been validated against actual final losses. Figure 6 shows the references drift down, but not that they are accurate. If the extrapolation misranks domains, the DRO weights will be miscalibrated, and the 2.7B domain-level deficit is consistent with that. This should be the referee's main push.

One smaller issue: the significance test uses five prompt variants from a single model, not multiple seeds. The p-values (0.046, 0.031) are marginal, and the paired t-test across tasks treats tasks as independent observations, which is questionable. The consistency across PPL, task averages, and win rates is more persuasive than the t-test.

This paper deserves a serious referee. It is a solid engineering contribution with a new combination and an honest controlled evaluation; it needs a revised abstract, a direct validation of the reference-loss extrapolation, and seed-level replication before publication.

Referee Report

4 major / 5 minor

Summary. The paper proposes DRPruning, a data-scheduling method for structured pruning and continued pretraining. It maintains per-domain validation losses, fits a four-parameter scaling law (Eq. 4) to predict the end-of-training loss as the DRO reference loss, and uses an EMA update (Eq. 6) to move the reference data ratio toward high-loss domains. The method is evaluated on Llama2-7B pruned to 1.3B/2.7B with SlimPajama data, on Qwen2-based multilingual models, and through instruction tuning and a GPT-constructed sentence-continuation benchmark. The paper claims lower perplexity, higher downstream scores, and better domain robustness than Sheared Llama-style scheduling and a controlled reproduction called ReSheared.

Significance. If the gains hold, the method is attractive: it adds no GPU overhead, releases code, and automates the two DRO hyperparameters that are normally hard to set. The controlled comparison against ReSheared and the pruning-phase improvements over the official Sheared Llama are concrete strengths, and the five-prompt significance analysis is a useful attempt at robustness checking. However, the headline claim that DRPruning 'surpasses similarly sized models' is not supported for continued pretraining at the 2.7B scale, and the scaling-law reference-loss mechanism is not validated against actual final losses. The paper's value currently rests on a fair-comparison baseline that may be weaker than the official Sheared Llama, so the scope of the claims needs to be narrowed or the missing evidence supplied.

major comments (4)

[Section 4.3, Table 2] The abstract's claim that DRPruning 'surpasses similarly sized models' in continued pretraining is contradicted by the paper's own data. For the 2.7B target, official Sheared Llama averages 52.53 on the 15-task suite, while DRPruning averages 51.63; on the domain-level benchmark in Section 5.2, official Sheared Llama scores 89.32 versus DRPruning's 86.00. The paper discloses the latter in Section 5.2, but the abstract and conclusion do not carry the necessary qualification. Please either restrict the headline claim to comparisons against ReSheared and other open models under identical data, or provide a 2.7B comparison under the official Sheared Llama data regime.
[Section 3.2, Eq. (4)] The reference loss, which is load-bearing for the DRO weights, is obtained by extrapolating a four-parameter scaling-law fit to partial loss curves, but the paper never validates the predicted end-of-training loss against the actual final per-domain loss. Figure 6 only shows that reference losses decrease gradually, which is a statement about sign, not about accuracy or calibration. This matters because Eq. (4) assumes a fixed training distribution, whereas DRPruning reweights the distribution every m steps based on these predictions. A concrete test would be to report predicted versus observed final per-domain losses under a fixed schedule, or to compare DRPruning with oracle reference losses computed from the actual final losses. Without such evidence, the mechanism's advantage over naive DRO remains unexplained.
[Appendix B.3] The claimed statistical significance (t=2.99, p=0.0032) is computed from five prompt variants per task on the same training run. These are not independent samples: task scores within a model are correlated, and prompt perturbation does not re-estimate model variance from independent training runs or seeds. The t-test should be run over independent training seeds or, failing that, over tasks with a mixed-effects model. In addition, the significance test compares DRPruning only with ReSheared, not with the official Sheared Llama model that the abstract claims to surpass.
[Section 5.3, Table 6] In the multilingual setting, the pruned-and-recovered model (Qwen2-7B to 1.8B with DRPruning, average 51.87) underperforms continued pretraining from the smaller unpruned Qwen2-1.5B with DRPruning (52.35). The paper acknowledges this in Section 5.3, but it limits the scope of the central 'pruning' claim: in this setting the benefit of DRPruning accrues to continued pretraining, not to pruning per se. Please state this qualification in the abstract and conclusion.

minor comments (5)

[Table 2] The column header '7B 2.7B 1.3B' does not match the actual column groups; please label the model-size groups explicitly above the Sheared/ReSheared/DRPruning columns.
[Appendix B.5] The phrase 'finer domain segmentatio' is missing the final 'n'.
[Eq. (6) and Appendix A.1] Equation (6) uses delta for the EMA update factor, while Appendix A.1 says 'lambda set to 0.1, which is also used for updating the reference data ratio.' Please align the notation.
[Figure 6] The caption contains a missing space in 'y = 0indicating no change'.
[Table 5] The 1.3B Constant baseline (55.29) is higher than both Sheared Llama (42.79) and ReSheared (47.64) on the generated continuation benchmark; please discuss whether the GPT-4-based filtering introduces a bias that favors the Constant schedule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DRPruning's central performance claims are tested on held-out tasks, and its scaling-law reference-loss estimator is an extrapolation from the same loss curves rather than a re-used input.

full rationale

The paper's derivation chain is not circular. The DRO weight update (Eq. 3), the scaling-law reference-loss prediction (Eq. 4), and the reference-ratio EMA (Eq. 6) form a feedback loop, but the quantities used to set the data schedule are per-domain validation losses, while the claims of success are evaluated on external held-out benchmarks: perplexity on validation data, 15 downstream tasks, instruction-tuning win rates, and multilingual task suites. The scaling-law fit extrapolates each domain's partial loss curve to the end of training; even if this extrapolation is unvalidated or unstable, that is a correctness or robustness concern, not circularity, because the predicted final loss is not defined as the current evaluation loss or as a parameter recovered from the target benchmark. The paper also self-reports negative results against official Sheared Llama 2.7B, noting that it scores 89.32 versus DRPruning's 86.00 on the authors' own domain benchmark; this is an explicit acknowledgment of a limitation rather than an attempt to force the central claim by construction. Self-citations, such as Deng et al. (2024) for the sentence-continuation benchmark construction and Jiao et al. (2022) for DRO settings, are not load-bearing to the main pruning-and-recovery derivation, and the scaling-law fitting procedure follows external work (Hoffmann et al., 2022a; Kaplan et al., 2020). No equation in the paper equates a fitted parameter with the predicted output in a way that would make the 'prediction' a rename of its input. Therefore, no specific circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies heavily on fitted scaling-law parameters and tuned DRO hyperparameters; the core contribution is a scheduling heuristic whose components are calibrated on the same training runs they control.

free parameters (6)

Scaling law coefficients A, E, alpha, beta per domain = Not reported (fit per domain via L-BFGS, best of three initializations)
Eqn 4 fits the model's own loss curve; these four parameters are refit after each evaluation and determine the reference loss used by DRO.
rho, chi-square ball radius = 0.1
Selected by preliminary experiments from {0.05, 0.1, 0.2}; controls how far the data distribution can move from the reference ratio.
delta, EMA update factor for reference ratio = 0.1
Controls how quickly the reference data ratio tracks the DRO output in Eqn 6.
Reference ratio bounds factor n = Number of domains (7 or 8)
The reference ratio is clipped to [p0/n, n*p0]; the bound range is a design choice.
Training stage thresholds = 20% (reference loss), 40% (reference ratio)
When the dynamic components are activated; chosen ad hoc without sensitivity analysis.
Loss smoothing lambda = 0.1
EMA coefficient for historical loss values in DRO weight computation, also used for reference ratio updates.

assumptions (5)

domain assumption Per-domain language modeling loss follows the scaling law l(P,T) = A/P^alpha * 1/T^beta + E (Eqn 4)
Used to extrapolate final loss from the first 20% of training; if the functional form is wrong, reference losses are biased.
domain assumption DRO with an f-divergence ball around a reference ratio and a reference loss produces good worst-case domain balance (Zhou et al. 2021)
Borrowed wholesale from prior DRO work; the paper adds only the dynamics of the reference loss and reference ratio.
ad hoc to paper The minimum predicted final loss is a good reference loss for accelerating convergence
Section 3.2 states the minimum is adopted to accelerate convergence; no theorem or ablation over min versus mean versus quantile is given.
ad hoc to paper Adjusting the reference ratio toward high-loss domains after 40% of training improves robustness without overfitting the highest-loss domain
Section 3.3; the bounds and activation schedule are justified only by stability arguments, not by formal analysis.
domain assumption Validation-set loss is a reliable per-domain performance proxy for downstream capabilities
The data ratios are updated from evaluation loss on a validation set; Section 5.2 tries to bridge this gap with a generated benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization." pith.science (2026). https://pith.science/paper/KTG4F5DK

@misc{pith2026241114055,
  author       = {Pith},
  title        = {Pith review of: DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KTG4F5DK}},
  note         = {Machine review of arXiv:2411.14055}
}

read the original abstract

Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. Experiments in monolingual and multilingual settings show that DRPruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. Further analysis demonstrates the robustness of DRPruning towards various domains and distribution shifts. Furthermore, DRPruning can determine optimal reference losses and data ratios automatically, suggesting potential for broader applications. Code and scripts are available at https://github.com/hexuandeng/DRPruning.

Figures

Figures reproduced from arXiv: 2411.14055 by the authors.

**Figure 2.** The curve of PPL changes during pruning from 7B. Over the first 640 iterations (the vertical dash line), the model size is gradually reduced from 7B to the target size, which causes an initial increase in PPL. Questions (NQ, Kwiatkowski et al., 2019) and TriviaQA (TriQA, Joshi et al., 2017). • We follow Pythia to report the 0-shot performance of LAMBADA (LAMB, Paperno et al., 2016), LogiQA (Liu et al., 2020), SciQ … view at source ↗

**Figure 3.** Win rate during instruction tuning. DRPrun [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: The generation procedure for our sentence [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Variation of data ratio, reference data ratio, and reference loss during continued pretraining. The top [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Convergence of masks over 3200 pruning steps. Similarity indicates the similarity between pruning decisions at a certain step and the final decisions at step 3200. “Layer”, “Hidden”, “Head”, and “Intermediate” correspond to the four pruning dimensions. To perform a d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[6]

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry

A framework for few-shot language model evaluation. Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2022. Scaling laws for neural machine translation. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net....

arXiv 2022
[7]

Stephanie Lin, Jacob Hilton, and Owain Evans

NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Assoc...

work page 2022
[8]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13865–13881, Miami, Florida, USA

Curriculum consistency learning for condi- tional sentence generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13865–13881, Miami, Florida, USA. Association for Computational Lin- guistics. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learn- ing efficien...

work page 2024
[10]

Pranav Rajpurkar, Robin Jia, and Percy Liang

Nemotron-4 15B Technical Report. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. Jun Rao, Xu...

work page 2018
[12]

In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing,...

work page 2024
[13]

You are a helpful assistant. Write a response that appropriately completes the request

OpenReview.net. Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1513–1528, Dublin, Ireland. Association for Computational Linguistics. An Yang, Baosong Yang, Binyuan Hui, Bo Zh...

work page 2021
[2014]

Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yao- jie Lu, Ben He, Shanshan Jiang, and Bin Dong

Data-Driven Robust Optimization. Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yao- jie Lu, Ben He, Shanshan Jiang, and Bin Dong

work page
[2016]

In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1119–1130, Austin, Texas

Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computa- tional Linguistics. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina To...

work page 2016

Show all 13 references

[2018]

In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings

Learning sparse neural networks through l_0 regularization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm...

2018
[2023]

Mathematical capabilities of chatgpt. In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Process- ing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Drew Fudenberg and David K Levine. 1998. The theory o...

2023
[2024]

Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. In Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evalua- tion, LREC/COLING 2024, 20-25 May, 20...

2024
[7439]

Su Lin Blodgett, Lisa Green, and Brendan O’Connor

AAAI Press. Su Lin Blodgett, Lisa Green, and Brendan O’Connor
[8740]

Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Sobol- eva, and Eric Xing

AAAI Press. Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Sobol- eva, and Eric Xing. 2023. SlimPajama-DC: Under- standing Data Combinations for LLM Training. Oleh Shliazhko, Alena Fenog...

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [6]

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry

A framework for few-shot language model evaluation. Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2022. Scaling laws for neural machine translation. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net....

arXiv 2022

[2] [7]

Stephanie Lin, Jacob Hilton, and Owain Evans

NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Assoc...

work page 2022

[3] [8]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13865–13881, Miami, Florida, USA

Curriculum consistency learning for condi- tional sentence generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13865–13881, Miami, Florida, USA. Association for Computational Lin- guistics. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learn- ing efficien...

work page 2024

[4] [10]

Pranav Rajpurkar, Robin Jia, and Percy Liang

Nemotron-4 15B Technical Report. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. Jun Rao, Xu...

work page 2018

[5] [12]

In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing,...

work page 2024

[6] [13]

You are a helpful assistant. Write a response that appropriately completes the request

OpenReview.net. Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1513–1528, Dublin, Ireland. Association for Computational Linguistics. An Yang, Baosong Yang, Binyuan Hui, Bo Zh...

work page 2021

[7] [2014]

Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yao- jie Lu, Ben He, Shanshan Jiang, and Bin Dong

Data-Driven Robust Optimization. Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yao- jie Lu, Ben He, Shanshan Jiang, and Bin Dong

work page

[8] [2016]

In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1119–1130, Austin, Texas

Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computa- tional Linguistics. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina To...

work page 2016