Pith. sign in

REVIEW 5 major objections 6 minor 62 references

SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SLM-Bench claims to be the first benchmark that scores small language models on correctness, computation, and environmental cost together, and it finds the accuracy leader is not the green leader.

desk verdict A useful SLM benchmarking artifact with real measurement weaknesses; the environmental rankings are not yet trustworthy, but the pipeline and trade-off data are worth engaging with. read the letter →

arxiv 2508.15478 v2 pith:ULQEZ6AJ submitted 2025-08-21 cs.CL cs.CYcs.PF

Nghiem Thanh Pham , Tung Kieu , Duc-Manh Nguyen , Son Ha Xuan , Nghia Duong-Trung , Danh Le-Phuoc This is my paper
classification cs.CLcs.CYcs.PF
keywords smalllanguagemodelsbenchmarkenvironmentalimpactenergyconsumptioncarbonemissionsmodelevaluationsustainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SLM-Bench is proposed as the first benchmark for small language models that scores them on correctness, computation, and consumption (cost, energy, CO2) at once. It runs 15 open models under 7 billion parameters across 9 tasks, 23 datasets, 14 domains, and 4 hardware settings, counting how often each model ranks first, second, or third. The headline finding is a split: Llama-3.2-1B wins the most accuracy medals, Phi-1.5B is the most energy-efficient model, and Mistral-7B offers the most balanced profile. A reader would care because model selection now has a quantitative basis for trading accuracy against sustainability in resource-constrained deployments.

What carries the argument

The load-bearing machinery is the benchmark itself: a standardized pipeline that funnels 23 datasets through 15 models, scored with 11 metrics organized into a three-axis taxonomy of correctness, computation, and consumption. Rankings are produced by medal counting: for every model-dataset-task-metric combination, first place earns gold, second silver, and third bronze. This aggregation is what turns many raw measurements into the paper's headline trade-off plots.

What would settle it

Run the full fine-tuning suite on identical hardware and compare each model's measured wall-plug energy and actual grid carbon intensity against the paper's estimates; if Phi-1.5B no longer wins the consumption medals, or if an accuracy leader also becomes efficient, the reported trade-off landscape changes.

Watch

Extended reading notes

Core claim

The paper's central claim is that SLM-Bench is the first systematic evaluation of small language models that puts environmental impact on equal footing with accuracy, and that the measurements expose stable trade-offs rather than a single winner. On the correctness axis, Llama-3.2-1B earns the most gold medals; on computation, GPT-Neo-1.3B leads; and on consumption, Phi-1.5B stands out as the most efficient. The authors further claim that correctness does not strongly correlate with computation or consumption, so parameter count alone predicts neither accuracy nor greenness; architecture, pretraining data, and implementation choices intervene.

Load-bearing premise

The sustainability rankings stand or fall on whether the software-estimated energy, CO2, and cloud-billed cost figures are accurate and comparable across all 15 models.

Editorial extensions

If this is right

  • Practitioners can now pick a small language model by stated priority: accuracy-first points to Llama-3.2-1B, efficiency-first to Phi-1.5B or GPT-Neo-1.3B, and balanced deployments to Mistral-7B.
  • Consumption should be reported alongside accuracy in future model comparisons, since the two can diverge sharply even for similarly sized models.
  • The relative rankings across hardware configurations suggest that model-level efficiency conclusions may transfer from server GPUs to edge devices, even though absolute magnitudes change.
  • The open pipeline lets new small language models be inserted and rescored without rebuilding the evaluation infrastructure.
  • Because inference happens far more often than fine-tuning, small per-inference efficiency differences can dominate deployment costs, as the paper's inference-cost appendix shows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its consumption rankings could serve as baselines for 'green SLM' claims by other developers, provided the estimation method is reproduced exactly.
  • Because the energy and CO2 figures are software estimates rather than wall-socket measurements, re-running the same suite with physical power meters is a natural extension; medal counts could shift, especially for models with bursty runtime behavior.
  • The decoupling of accuracy from model size suggests that pretraining data quality and recipe may be a stronger lever than parameter count for small models, a hypothesis worth testing with controlled pretraining on a fixed data budget.
  • A testable extension would be to report confidence intervals or measurement uncertainty for each consumption metric, so model selection is not driven by small differences between estimates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. SLM-Bench introduces a benchmark specifically for small language models, evaluating 15 SLMs across 23 datasets, 9 task types, 4 hardware configurations, and 11 metrics spanning correctness, computation, and consumption (cost, energy, CO2 emissions). The authors propose a medal-counting ranking over all metric–dataset pairs and report that Llama-3.2-1B leads in correctness, GPT-Neo-1.3B in computation, and Phi-1.5B in energy efficiency. The paper claims to be the first comprehensive SLM benchmark that jointly measures correctness and environmental impact, and it provides an open-source pipeline and a public leaderboard. The extended version adds appendices on datasets, models, metrics, and an inference-cost table.

Significance. If the environmental measurements are credible, SLM-Bench would be a useful resource for model selection in resource-constrained settings, addressing a genuinely underexplored dimension of SLM evaluation. The breadth of models, datasets, tasks, and hardware configurations is a strength, and the open-source pipeline and leaderboard are reproducible contributions. The paper also makes an explicit and falsifiable claim about trade-offs among correctness, computation, and consumption—for example, that Llama-3.2-1B is the most accurate but not the most efficient, while Phi-1.5B is the most energy-efficient. These claims are useful to practitioners. However, the environmental dimension—the paper's distinctive novelty—currently rests on undocumented estimator assumptions, and the ranking methodology contains arbitrary aggregation choices. The contributions are therefore potentially significant, but they require additional validation and disclosure before the main claims can be taken as established.

major comments (5)
  1. [Section 3.3 and Section 6] The environmental metrics are load-bearing for the paper's central novelty, yet they are not reproducible or validated. Section 3.3 says energy is measured via the Zeus package and CO2 via ML CO2 Impact, but the power-draw model, PUE, carbon-intensity default, electricity price, and any model-specific parameters are never disclosed. Section 6 admits these are 'estimated using standardized formulas rather than real-time measurements.' Since Figure 4c and Table A4 drive the claim that Phi-1.5B is the most energy-efficient model (Sections 4.4 and 4.6), a miscalibrated default for the NVIDIA L4 GPU or the wrong grid region could change the consumption rankings. The authors should report the exact package versions, all estimator parameters, and a sensitivity analysis over plausible PUE/carbon-intensity values; ideally they should also compare against at least one wall-socket measurement.
  2. [Section 3.3 vs. Appendix F] There is a direct contradiction about whether inference is part of the benchmark. Section 3.3 states: 'For inference, we did not include runtime comparisons in the paper, as the differences between models were insignificant.' Yet Appendix F and Table A4 present per-1,000-token inference cost, energy, CO2, FLOP, and runtime, and the text concludes that 'Phi-1.5B demonstrates the best performance in both computation efficiency and energy consumption.' The main text also does not mention that Table A2's throughput/latency columns appear to come from vendor documentation rather than from the authors' measurements. The authors must clarify which numbers were measured by them, which were taken from external sources, and why the main text excludes inference while an appendix asserts a ranking based on it.
  3. [Section 4.1] The paper makes an important generalization claim: 'while the magnitudes of the results vary on different hardware configurations, the relative ranking among the SLMs remains consistent.' However, results for the A10, Jetson Orin AGX 16GB, and Jetson Orin AGX 64GB configurations are not shown in the paper and are only said to be available on the leaderboard. Because the benchmark's stated goal is to inform deployment on diverse hardware, this claim is checkable only by going to an external website, which may not be stable or archived. The authors should include at least a summary table or appendix showing the relative rankings for all four configurations, or soften the claim to what is actually presented.
  4. [Section 3.5 and Section 4.5] The headline ranking depends on an arbitrary aggregation over heterogeneous metrics. The medal-counting procedure in Section 3.5 counts every metric–dataset pair equally, but many are redundant: for text-generation datasets, BLEU, ROUGE, METEOR, and Perplexity are all computed on the same outputs, and accuracy and F1 are similarly redundant on classification datasets. The 3/2/1 gold/silver/bronze weights in Section 4.5 are chosen without justification. The statement that Llama-3.2-1B 'outperforms the other SLMs' (Section 4.3) could change if metrics were grouped differently or weights were varied. The authors should provide robustness checks, such as per-dataset averaging, normalized-score aggregation, or at least a table showing how the ranking changes under alternative weighting schemes.
  5. [Section 4.2] The hyperparameter settings are underreported, which affects both the correctness ranking and the consumption ranking, since fine-tuning time/cost depends on epochs, batch size, and LoRA configuration. Section 4.2 says that when values are not available from existing sources, the authors perform random search over learning rate, batch size, epochs, LoRA rank, and dropout, but it does not report the chosen values per model–dataset pair, the search budget, the validation metric, or the seed. Without this, the benchmark results are not fully reproducible, and it is unclear whether differences in accuracy reflect model quality or hyperparameter tuning effort. The authors should release a configuration table or configuration file for every run.
minor comments (6)
  1. [Table 2 vs. Table A2] Phi-1.5B is listed with 1.42B parameters in Table 2 but with 2.70B parameters in Table A2. This is a factual inconsistency that must be fixed; the same row also shows a training time of '12 days (32 GPUs)' that is not consistent with the model's known release documentation.
  2. [Figure 1] The timeline text is garbled: '08/202303/2021' appears without a separator, making the release dates impossible to read.
  3. [Section 2.1] The claim that 'Existing benchmarks, such as GLUE and SuperGLUE, primarily evaluate LLMs' is inaccurate: GLUE and SuperGLUE were designed for pretrained encoders such as BERT, not for autoregressive LLMs. This does not affect the main results but should be corrected.
  4. [Section 3.2] The inclusion criterion is 'fewer than 7 billion parameters,' yet Mistral-7B and Zephyr-7B are listed with 7.00B parameters. The criterion should state 'at most 7B' or the boundary should be clarified.
  5. [Section 4.4] There is a typo: 'LLaMA-2.7B' should be 'LLaMA-2-7B'.
  6. [References] The reference for the Big-Bench paper is listed as 'Big-Bench Authors. 2023,' but the in-text citation is '(Authors, 2023),' which does not match. Please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SLM-Bench is an empirical measurement study; its rankings follow from defined metrics and are not fitted to their own outputs.

full rationale

SLM-Bench reports measurements of 15 SLMs on 23 datasets over four hardware configurations. The central results (Llama-3.2-1B most correct, Phi-1.5B most energy-efficient, etc.) are produced by a transparent ranking rule (Section 3.5: medal counts over K×N×T×M cases; Section 4.5: 3/2/1 weighted scores). These are author-defined aggregation choices, not derivations from first principles; no parameter is fitted to a target result and no equation is defined in terms of the outcome it is said to predict. Consumption metrics are obtained from platform billing and from Zeus/ML CO2 Impact estimation packages (Section 3.3). Section 6 explicitly concedes energy and CO2 are estimated by standardized formulas rather than real-time measurements; this is an acknowledged measurement-validity limitation, not a circularity, because the estimates are external inputs rather than consequences of the benchmark's own conclusions. No load-bearing self-citation appears: the reference list contains no prior work by the present authors invoked to justify the benchmark's design or to exclude alternatives. The 'first benchmark' novelty claim is an absence claim argued from the related-work survey, not a circular reduction. Accordingly, no circular step exists; score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's contributions rest on standard ML benchmarking practice plus tool-based estimates. The main unpaid assumptions are (1) that energy/CO2 estimates from Zeus and ML CO2 Impact with undisclosed parameters reflect real consumption, (2) that per-model hyperparameter tuning was fair, and (3) that L4 results are representative of all four hardware platforms. No mathematical derivation pulls constants from fits; the 'free parameters' are methodological choices (hyperparameters, medal weights, tool defaults) that shape the rankings.

free parameters (3)
  • Per model-dataset hyperparameters (learning rate, batch size, epochs, LoRA rank, dropout) = not reported; chosen via random search over stated ranges
    Section 4.2 selects these per model-dataset, so differences in tuning effort or luck can change medal counts; results depend on choices not fixed by the paper.
  • Medal weights (gold 3, silver 2, bronze 1) = 3, 2, 1
    Section 4.5 introduces a score-based aggregation with arbitrary weights; different weights would change the trade-off conclusions.
  • Energy/CO2 model parameters inside Zeus and ML CO2 Impact (power draw, PUE, carbon intensity) = not disclosed
    Sections 3.3 and 6; the consumption rankings are determined by these package defaults, which the paper does not specify.
assumptions (5)
  • domain assumption The 23 datasets are valid representatives of their assigned tasks/domains and are loaded correctly
    Section 3.1; the paper imports datasets from HF and prior leaderboards; several dataset citations are wrong (AQuA, GSM8k, CommonsenseQA), which weakens the assumption.
  • domain assumption The energy and CO2 estimates from Zeus and ML CO2 Impact approximate true consumption
    Sections 3.3 and 6; the paper concedes these are standardized formula estimates, not sensor measurements.
  • domain assumption Relative rankings on the NVIDIA L4 GPU are representative of the other three hardware configs
    Section 4.1 asserts 'the relative ranking among the SLMs remains consistent' without presenting the other hardware results in the paper.
  • domain assumption LoRA fine-tuning is an appropriate and fair way to assess SLMs across heterogeneous tasks
    Section 4.2; all models are fine-tuned with LoRA, but the paper does not state that LoRA configuration parity holds across models (e.g., targeted modules).
  • ad hoc to paper Medal counting over heterogeneous metric-dataset pairs yields a meaningful overall ranking
    Section 3.5; the aggregation counts many metrics of different types equally (accuracy medals and cost medals), a choice specific to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version." pith.science (2026). https://pith.science/paper/ULQEZ6AJ

@misc{pith2026250815478,
  author       = {Pith},
  title        = {Pith review of: SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULQEZ6AJ}},
  note         = {Machine review of arXiv:2508.15478}
}
read the original abstract

Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.

Figures

Figures reproduced from arXiv: 2508.15478 by the authors.

Figure 1
Figure 1. Overview of SLM-Bench comprehensively assesses the performance of SLMs across diverse tasks while also quanti￾fying their environmental impacts. This lack of standardized evaluation hinders a deeper understanding of their practical implications, particularly in resource-constrained environ￾ments where efficiency and sustainability are paramount. Addressing this gap is essential to unlock the full potential of SLMs a… view at source ↗
Figure 2
Figure 2. Benchmarking Pipeline 3.5 Ranking Methodology We propose a ranking method for SLMs based on their performance across multiple datasets and evaluation metrics. This method counts how often each model ranks first, second, or third in different experimental settings. Specifically, let K be the number of SLMs, N the number of datasets, T the number of tasks, and M the num￾ber of evaluation metrics. For each model k, we … view at source ↗
Figure 3
Figure 3. Overall Results rations use NVIDIA L4 and A10 GPUs, while the edge-device configurations use NVIDIA Jetson Orin AGX with 16GB and 64GB memory, respec￾tively. Due to space constraints, the main paper reports results only on the NVIDIA L4 GPU. Re￾sults for the other configurations are provided on the Leaderboad. Furthermore, while the magni￾tudes of the results vary on different hardware configurations, the relative r… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Taxonomy Results for specific applications. Appendix E further discusses the metric contribution to medals in 15 models. Further, Appendix F presents the inference cost. 4.4 Evaluation Taxonomy We categorize the evaluation metrics into three distinct types: correctness…
Figure 5
Figure 5. Figure 5: Performance Trade-off, only Gold Medals Correctness Computation Consumption Llama-3.2-1B GPT-Neo-1.3B Zephyr-7B Phi-1.5B Mistral-7B [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance Trade-off, Score-based Evalua [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 48 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Al - Shaibani and Irfan Ahmad

    Maged S. Al - Shaibani and Irfan Ahmad. 2023. Consonant is all you need: a compact representation of english text for efficient NLP . In Findings of the Association for Computational Linguistics (EMNLP), pages 11578--11588

    work page 2023

  4. [4]

    Big-Bench Authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transaction of Machine Learning Research, 2023

    work page 2023

  5. [5]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 7432--7439

    work page 2020

  6. [6]

    Quentin Brabant, Gw \' e nol \' e Lecorv \' e , and Lina Maria Rojas - Barahona. 2022. Coqar: Question rewriting on coqa. In Proceedings of the Language Resources and Evaluation Conference ( LREC ) , pages 119--126

    work page 2022

  7. [7]

    Bursztyn, David Demeter, Doug Downey, and Larry Birnbaum

    Victor S. Bursztyn, David Demeter, Doug Downey, and Larry Birnbaum. 2022. Learning to perform complex tasks through compositional fine-tuning of language models. In Findings of the Association for Computational Linguistics (EMNLP), pages 1676--1686

    work page 2022

  8. [8]

    Christopher Clark, Kenton Lee, Ming - Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2924--2936

    work page 2019

Show all 62 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457

    2018 arXiv

  2. [10]

    Payal Dhar. 2020. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2(8):423--425

    2020

  3. [11]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the Conference of the North American Chapter of the Association for Comput...

    2019

  4. [12]

    Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG challenge. In Proceedings of the International Conference on Natural Language Generation (INLG), pages 322--328

    2018

  5. [13]

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. 2024. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  6. [14]

    Ze - Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao, and Ji - Rong Wen. 2023. Small pre-trained language models can be fine-tuned as large models via over-parameterization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 3819--3834

    2023

  7. [15]

    Gowda, Xinyue Hao, Gen Li, Laura Sevilla - Lara, and Shashank Narayana Gowda

    Shreyank N. Gowda, Xinyue Hao, Gen Li, Laura Sevilla - Lara, and Shashank Narayana Gowda. 2023. Watt for what: Rethinking deep learning's energy-performance relationship. CoRR, abs/2310.06522

    2023 arXiv

  8. [16]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  9. [17]

    Rashmi Gupta, Tushar Agarwal, Aviral Mehlawat, Deeksha Mathur, Devendra Kumar Somwanshi, and Anil Kumar. 2023. SMER: A novel medical entity recognition technique based on scispacy (bc5cdr) and med7. In Proceedings of the International Conference on Information Management & Mac...

    2023

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2022

  11. [19]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tiny BERT : Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics (EMNLP), pages 4163--4174

    2020

  12. [20]

    Juraj Juraska, Kevin Bowden, and Marilyn A. Walker. 2019. Viggo: A video game corpus for data-to-text generation in open-domain conversation. In Proceedings of the International Conference on Natural Language Generation (INLG), pages 164--172

    2019

  13. [21]

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09700

    2019 arXiv

  14. [22]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In Proceedings of the Conference on Empirical Methods in Natural Language Processin (EMNLP), pages 785--794

    2017

  15. [23]

    Lewis, Yiming Yang, Tony G

    David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361--397

    2004

  16. [24]

    Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Zhengliang Liu, Zihao Wu, Peng Shu, Jie Tian, Tianze Yang, Shaochen Xu, Yanjun Lyu, Parker Blenk, Jacob Pence, Jason Rupram, Eliza Banu, Ninghao Liu, Linbing Wang, Wen - Zhan Song, Xiaoming Zhai, Kenan Song, Dajiang Zhu, Beiwen Li, X...

    2024 arXiv

  17. [25]

    Bingbin Liu, S \' e bastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. 2023. Tinygsm: Achieving 80 \ CoRR, abs/2312.09241

    2023 arXiv

  18. [26]

    Vivian Liu and Yiqiao Yin. 2024. Green AI: exploring carbon footprints, mitigation strategies, and trade-offs in large language model training. Discovery Artificial Intelligence, 4(1):49

    2024

  19. [27]

    Friedler, and Jonathan P

    Kadan Lottick, Silvia Susai, Sorelle A. Friedler, and Jonathan P. Wilson. 2019. Energy usage reports: Environmental awareness as part of algorithmic accountability. CoRR, abs/1911.08354

    2019 arXiv

  20. [28]

    Alexandra Sasha Luccioni, Sylvain Viguier, and Anne - Laure Ligozat. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24:253:1--253:15

    2023

  21. [29]

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo, and Xipeng Qiu. 2024. Full parameter fine-tuning for large language models with limited resources. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 8187--8198

    2024

  22. [30]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. In Proceedings of the Annual Conference on Neural Information Processing Systems ( NeurIPS )

    2023

  23. [31]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Ad \' a mek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1773--1781

    2023

  24. [32]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2381--2391

    2018

  25. [33]

    Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. CoRR, abs/2406.07545

    2024 arXiv

  26. [34]

    Eklund, and Jinman Kim

    Usman Naseem, Imran Razzak, Matloob Khushi, Peter W. Eklund, and Jinman Kim. 2021. Covidsenti: A large-scale benchmark twitter data set for COVID-19 sentiment analysis. IEEE Transactions on Computational Social Systems , 8(4):1003--1015

    2021

  27. [35]

    OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774

    2023 arXiv

  28. [36]

    Patterson, Joseph Gonzalez, Quoc V

    David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis - Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. CoRR, abs/2104.10350

    2021 arXiv

  29. [37]

    Yuchen Peng, Ke Chen, Lidan Shou, Dawei Jiang, and Gang Chen. 2023. AQUA: automatic collaborative query processing in analytical database. Proc. VLDB Endow. , 16(12):4006--4009

    2023

  30. [38]

    Soham Poddar, Paramita Koley, Janardan Misra, Niloy Ganguly, and Saptarshi Ghosh. 2025. Towards sustainable NLP: insights from benchmarking inference energy in large language models. CoRR, abs/2502.05610

    2025 arXiv

  31. [39]

    Sumanth Prabhu, Aditya Kiran Brahma, and Hemant Misra. 2022. Customer support chat intent classification using weak supervision and data augmentation. In Proceedings of the Joint International Conference on Data Science & Management of Data (CODS-COMAD) , pages 144--152

    2022

  32. [40]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 8732--8740

    2020

  33. [41]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distil BERT , a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108

    2019 arXiv

  34. [42]

    Or Sharir, Barak Peleg, and Yoav Shoham. 2020. The cost of training NLP models: A concise overview. CoRR, abs/2004.08900

    2020 arXiv

  35. [43]

    Dong Shu and Mengnan Du. 2024. Comparative analysis of demonstration selection algorithms for LLM in-context learning. CoRR, abs/2410.23099

    2024 arXiv

  36. [44]

    Aditi Singh, Nirmal Prakashbhai Patel, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2024. A survey of sustainability in large language models: Applications, economics, and challenges. CoRR, abs/2412.04782

    2024 arXiv

  37. [45]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP . In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 3645--3650

    2019

  38. [46]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , pages 13693--13696

    2020

  39. [47]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  40. [48]

    Adrian Theuma and Ehsan Shareghi. 2024. Equipping language models with tool use capability for tabular data analysis in finance. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 90--103

    2024

  41. [49]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA : Open and effic...

    2023 arXiv

  42. [50]

    David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George F. Foster. 2023. Prompting palm for translation: Assessing strategies and performance. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 15406--15427

    2023

  43. [51]

    Lulu Wan, George Papageorgiou, Michael Seddon, and Mirko Bernardoni. 2019. Long-length legal document classification. CoRR, abs/1912.06905

    2019 arXiv

  44. [52]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 a . SuperGLUE : A stickier benchmark for general-purpose language understanding systems. In Proceedings of the Annual Conference on Neural Informati...

    2019

  45. [53]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 b . GLUE : A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations, (ICLR)

    2019

  46. [54]

    Xiaorong Wang, Clara Na, Emma Strubell, Sorelle Friedler, and Sasha Luccioni. 2023. Energy and carbon considerations of fine-tuning BERT . In Findings of the Association for Computational Linguistics ( EMNLP ) , pages 9058--9069

    2023

  47. [55]

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. 2024. Qa-lora: Quantization-aware low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  48. [56]

    Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024 a . MentaLLaMA : Interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference (WWW) , pages 4489--4500

    2024

  49. [57]

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024 b . Do large language models latently perform multi-hop reasoning? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 10210--10229

    2024

  50. [58]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 4791--4800

    2019

  51. [59]

    Lihui Zhang and Ruifan Li. 2023. Knowledge prompting with contrastive learning for unsupervised commonsenseqa. In Neural Information Processing - Proceedings of the International Conference on Neural Information Processing ( ICONIP ) , pages 27--38

    2023

  52. [60]

    Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024 a . Dual-space knowledge distillation for large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 18164--18181

    2024

  53. [61]

    Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. 2024 b . Plug-and-play: An efficient post-training pruning method for large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2024

  54. [62]

    Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, and Haibin Chen. 2023. Preserving commonsense knowledge from pre-trained language models via causal inference. In Proceedings of the Annual Meeting of the Association for Computa...

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.