Pith. sign in

REVIEW 5 major objections 7 minor 32 references

CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned small models rival much larger LLMs while emitting far less carbon, and CEGI quantifies the trade-off.

desk verdict The raw emissions measurements are a useful data point, but CEGI does not reproduce from its own equations and its percentage-gain normalization rewards low baselines, so the central metric needs a major rework before the trade-off conclusions can be trusted. read the letter →

arxiv 2412.02602 v1 pith:LKEVGUJH submitted 2024-12-03 cs.CL cs.AIcs.CYcs.LG

Abhas Kumar , Kapil Pathak , Rajesh Kavuru , Prabhakar Srinivasan This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG
keywords carbon-efficientgainindexsmalllanguagemodelsvisionlow-rankadaptationquantizationcarbonemissionssustainableAImodelselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that for image captioning, visual question answering, dialogue summarization, and text-to-SQL, small language models and vision-language models fine-tuned with LoRA can reach scores close to those of much larger models while emitting far less carbon. To make this comparison quantitative, the authors define CEGI, the carbon emission per unit percentage gain per million trainable parameters. Across their experiments, smaller fine-tuned variants such as Qwen2.5-0.5B and Qwen-VL-2B achieve most of the accuracy of the largest tested models, and the metric identifies the same sweet-spot models that a human would choose from the plots. The paper concludes that the marginal accuracy gained from larger models is not worth the extra emissions, and that 4-bit and 8-bit quantization lower energy use without hurting scores. A sympathetic reader would take this as an argument to prefer small, task-tuned, quantized models when both accuracy and environmental cost matter.

What carries the argument

The load-bearing object is the CEGI score, defined as $G^{o}_{M,\mu,T_p} = \frac{(\sum CE) \times |L_r|}{(\sum G_{M,\mu}(FT,BM)) \times (\sum T_p)}$, giving grams of CO2 per percentage point of gain per million trainable parameters. The other two mechanisms feed into it: LoRA keeps the trainable-parameter denominator small and task-specific, and 4-bit and 8-bit quantization cut the energy use that appears in the numerator while leaving scores nearly unchanged. The identity works because a lower CEGI means a model buys more performance per gram of carbon per parameter.

What would settle it

Recompute CEGI for the paper's own tables using absolute score differences rather than percentage gains, or add a small floor to near-zero baselines such as the 0.0037 BLEU score; if the lowest-CEGI model changes for any of the four tasks, the ranking is an artifact of percentage scaling rather than a stable measure of efficiency.

Watch

Extended reading notes

Core claim

The authors claim that fine-tuned SLMs and VLMs deliver performance comparable to LLMs at substantially lower carbon cost, and that CEGI orders models by that trade-off. For example, in dialogue summarization Qwen2.5-0.5B reaches a ROUGE-1 of 0.42 versus 0.51 for Qwen2.5-14B while emitting about 25 g instead of about 154 g of CO2; on image captioning Qwen-VL-2B reaches 0.35 SPICE versus 0.31 for Llama-3.2-11B while emitting 97 g instead of 222 g. The lowest-CEGI model is Qwen-VL-7B for both image captioning and visual QA, Llama-3.2-1B for dialogue summarization, and Qwen2.5-3B for text-to-SQL, and these are the same models the authors mark as the performance-emissions sweet spots. On this evidence, the paper asserts that for these tasks larger models' incremental accuracy does not justify their disproportionate emissions.

Load-bearing premise

CEGI assumes that a percentage gain over the base model is a fair, comparable measure across metrics with very different scales and saturation points, and that dividing total emissions by that percentage and by trainable parameters yields a meaningful efficiency ranking.

Editorial extensions

If this is right

  • A practitioner working on these four tasks can pick a fine-tuned 0.5B or 2B model and get most of the accuracy of a 7B, 11B, or 14B model while cutting measured carbon emissions by more than half.
  • Lower-bit quantization at 4-bit and 8-bit becomes a safe default: performance varies by less than a couple of percent while emissions drop, so efficiency gains do not have to be traded against accuracy.
  • CEGI gives a single number for ranking candidate models on performance-per-carbon, which could be adopted in model-selection workflows whenever emission measurements are available.
  • The reported pattern implies that model families should be evaluated at their small end before scaling up, because the largest variants' accuracy gains over the small variants are small relative to their emissions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an editorial extension, CEGI's percentage-gain normalization is risky for metrics with near-zero baselines, so a robust version of the index might use absolute gains, a floor, or a saturating transformation before it is used as a general benchmark.
  • The paper measures emissions mainly from fine-tuning on one GPU; extending the same index to inference, deployment, and differing hardware would test whether the ranking persists outside the lab.
  • A testable extension would be to recompute CEGI on a task where the base model is already strong; if rankings invert, the index is rewarding models that start weak rather than models that are genuinely efficient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces CEGI (Carbon Efficient Gain Index), a metric that combines carbon emissions, percentage performance gain over a base model, and trainable-parameter count, to rank fine-tuned small language models (SLMs) and vision-language models (VLMs) against larger models across four tasks: image captioning, visual question answering, dialogue summarization, and text-to-SQL. The authors fine-tune Qwen and LLaMA family models with LoRA at ranks 4, 8, 16, and 32 and at 4-bit and 8-bit quantization, measure carbon footprints with eco2AI, and report performance and emissions for each configuration. They conclude that fine-tuned SLMs and VLMs can achieve performance comparable to larger models at substantially lower carbon cost, and that CEGI identifies the optimal model in each task consistently with human judgment.

Significance. If the CEGI metric were valid, it would provide a practically useful tool for carbon-aware model selection. The empirical direction—showing that small fine-tuned models can reach near-large-model performance at lower emissions—is timely and relevant for the Green AI community. The paper covers a diverse set of tasks and model families, and the use of eco2AI for emissions tracking is a positive step toward reproducible carbon accounting. The main contribution, however, is the metric itself, and the construction and validation of that metric currently have serious problems that undermine the paper's central claims.

major comments (5)
  1. [Section 6.3, Eq. (4)-(10), Tables 7 and 11] The base-relative percentage gain of Eq. (4) is scale-dependent and not comparable across metrics. In Visual-QA, Table 7 shows base BLEU values of 0.0010-0.0037, so fine-tuning yields percentage gains of 1,500-7,570% and near-zero CEGI values in Table 11. Changing the base BLEU of Qwen-VL-7B from 0.0010 to 0.0020 would reduce the reported 7,570% gain to about 3,785%, roughly doubling the CEGI. This means the metric rewards low baseline scores rather than absolute performance improvement, and the CEGI rankings that support the model-selection conclusions are not robust to small changes in the input data.
  2. [Section 6.3, Eqs. (9)-(10), Tables 10-13 and Table 20] The CEGI values reported in Tables 10-13 are not reproducible from the stated formulas and the data provided. For image captioning, Qwen-VL-7B has total CE=137.97g, average gain=166.67%, and trainable parameters from Table 20 (averaged over Lr=8,16,32 gives 3.12m; summed gives 9.37m); substituting these into Eq. (10) yields CEGI≈0.265, but Table 10 reports 0.36. Similar discrepancies appear for Qwen-VL-2B in Table 10 and Qwen-VL-7B in Table 11. Additionally, the 'Train params' column in Table 12 for Qwen2.5-0.5B is 0.36m, which matches neither the sum (1.26m) nor the average (0.42m) of the Table 20 values for Lr=4,8,16 used in the dialogue summarization experiments. The rankings and the practical recommendations built on them cannot be verified without a correct and consistent computation.
  3. [Section 6.3, final paragraph] The claim that the lowest-CEGI model 'consistently identifies the same model selected through human judgment' is unsupported. No description of the human-selection procedure, criteria, or judgment data is provided, so this validation is circular: CEGI is computed from the same performance and emissions measurements that would also inform any human selection, and there is no independent evidence that CEGI matches human judgment. The authors need to specify the human-selection protocol and provide the resulting choices for each task.
  4. [Section 5, opening paragraph] The paper states that every reported value, including performance metrics and emissions, is the mean of 5 independent runs, but no variance, standard deviation, or confidence interval is reported anywhere. Given that the CEGI differences between models are sometimes very small (e.g., 0.005 vs 0.034 in Table 11) and that ROUGE-1 scores are often tied at 0.50-0.51, the reader cannot determine whether the claimed trade-offs and rankings are statistically meaningful. Error bars or per-run values should be provided for at least the key tables.
  5. [Sections 5.1, 5.2, and 6.1] The GPT-4o baseline is not a matched comparison: it is evaluated zero-shot, with no fine-tuning, no carbon-emission measurement, and a different prompt setup. The claim that 'fine-tuned models surpass GPT-4o' (Section 6.1.1) therefore compares adapted small models to an unadapted generalist, not to an equivalently fine-tuned LLM. This weakens the abstract's broader conclusion that fine-tuned SLMs and VLMs achieve performance 'comparable to LLMs,' since the comparison does not isolate model size as the variable of interest.
minor comments (7)
  1. [Abstract vs. Section 7] The metric is called 'Carbon Efficient Gain Index' in the abstract and Section 6.3, but 'Carbon Efficient Emission Index' in Section 7; the terminology should be made consistent throughout.
  2. [Table 1] The row label 'Test-to-SQL' is a typo for 'Text-to-SQL', and the Visual QA metric is written as 'BLUE' instead of 'BLEU'.
  3. [Section 6.2] The phrase 'with lower emmisions compared to Qwen2.5-14B' contains a spelling error ('emmisions' should be 'emissions').
  4. [Section 6.1.1] The sentence 'the percentage improvement is more significant in smaller models (e.g., Qwen-VL-7B's 171.43% vs. Llama-3.2-11B's 72.22%)' contrasts models that differ in architecture and family, not just size; a within-family comparison (e.g., Qwen-VL-2B vs Qwen-VL-7B) would be clearer.
  5. [Section 7, Limitations] The limitation statement reads 'our focus on Qwen and Llama variants may represent the broader landscape of model architectures,' but the intended meaning appears to be 'may not represent'; the missing 'not' changes the sentence into the opposite of what is likely meant.
  6. [Table 3 and Table 7] Table 3 lists a GPT-4o BLEU value of 0.0013 for Qwen-VL-2B, but Table 7 computes gains versus GPT-4o using GPT-4o BLEU = 0.0037; the inconsistency should be resolved.
  7. [Figures 2 and 6] The Visual-QA performance plots use an exponential or log scale for BLEU scores, as indicated by the 'e3x scale' label in Figure 2, but the captions do not explain that the apparent visual gaps correspond to very different absolute differences; this should be stated explicitly.

Circularity Check

1 steps flagged · score 4.0 of 10

CEGI's validation against 'human judgment' is circular because the starred optimal models and the low-CEGI picks encode the same author-defined trade-off; the main empirical efficiency claims rest on direct measurements and are not circular.

  1. self definitional [Section 6.2 (star definition) and Section 6.3 (CEGI/human-judgment alignment)]
    "The model marked with the '⋆' symbol in plots 1, 2, 7, and 8 represents the optimal model, achieving maximum performance with minimum carbon emissions. ... Furthermore, the model with the lowest Carbon Efficient Gain Index (CEGI) score across all four tasks ... consistently identifies the same model selected through human judgment, which is also marked with the '⋆' symbol in plots 5, 6, 7, and 8."

    The paper's claimed external validation of CEGI is self-referential. Section 6.2 defines the starred model as 'the optimal model, achieving maximum performance with minimum carbon emissions,' i.e., the same trade-off objective that CEGI formalizes. Section 6.3 then reports that the lowest-CEGI model 'consistently identifies the same model selected through human judgment' and cites this alignment as evidence that CEGI 'is well-correlated with human judgment' and 'reliable.' No independent human-judgment protocol, rubric, or external raters are provided; the star markers are the authors' own selections from the same data.

full rationale

The main derivation chain (measure base/fine-tuned performance, measure carbon emissions, define CEGI as emissions per unit percentage gain per trainable parameter) is not circular: it is an explicitly proposed operationalization, and the paper's headline conclusions about small-model efficiency are supported by direct pairwise comparisons of measured SPICE/BLEU/ROUGE/EA and emissions, which do not depend on CEGI. There are no load-bearing self-citations, no imported uniqueness theorem, and no renamed known result. The one genuine circularity is the validation claim in Section 6.3: the 'human judgment' used to validate CEGI is the same authors' starred 'optimal model' selections, defined by the same maximum-performance/minimum-emissions objective that CEGI encodes, so the reported agreement is self-definitional. Concerns about the metric's scale-dependence (percentage gains near zero base BLEU, aggregation across incommensurable metrics, and the non-reproducible train-parameter normalization in Tables 10-13) are substantive validity/correctness issues but are not circularity. Hence a moderate score of 4 for the self-referential validation, while the core empirical claim remains independently evidenced.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims rest on the eco2AI emission model, the comparability of percentage gains across disparate metrics, and the validity of CEGI as an efficiency measure. No physical constants or external benchmarks are used; all quantities are measured or chosen by the authors. The metric itself is an invented ledger entry with no independent evidence.

free parameters (5)
  • LoRA scaling parameter alpha = 16
    Chosen by hand in Section 5; controls the magnitude of low-rank updates and affects performance and emissions.
  • LoRA dropout rate = 5%
    Set in Section 5 to introduce regularization; not varied and not optimized.
  • Number of fine-tuning epochs = 1
    Forced to 1 epoch for all tasks to ensure uniformity; chosen by hand, not optimized.
  • Emission intensity coefficient gamma = not specified
    Taken from eco2AI based on region, but the region and value are not reported, affecting all carbon emission numbers.
  • Power usage effectiveness (PUE) = not specified
    Included in Eq. 1 from eco2AI, but not reported for the hardware used, affecting the absolute CE values.
assumptions (4)
  • domain assumption The eco2AI formula CF = gamma * PUE * (E_CPU + E_GPU + E_RAM) accurately estimates carbon emissions from measured power consumption.
    Section 3.1 adopts this model without validating it against a physical power meter. All carbon emission comparisons rest on this formula.
  • ad hoc to paper Percentage gain calculated in Eq. 4 is a meaningful, scale-free measure of performance improvement that is comparable across vastly different metrics.
    Used to define average gain and CEGI; not justified against near-zero baselines or metric saturation, which can distort percentage gains.
  • domain assumption Zero-shot GPT-4o evaluations are an appropriate baseline for comparing fine-tuned models.
    GPT-4o is used as a reference, but it is not fine-tuned on the task datasets, making the comparison asymmetric and favoring the fine-tuned models.
  • domain assumption The selected task metrics (SPICE, BLEU, ROUGE, EA/VES) adequately measure task performance.
    Standard metrics are used without discussing their limitations for the specific small datasets, such as floor and ceiling effects.
invented entities (1)
  • CEGI (Carbon Efficient Gain Index)
    purpose: To provide a normalized measure of carbon emissions per unit percentage gain per million trainable parameters for model selection.
    The metric is introduced in Section 6.3 and validated only against the authors' own 'human judgment' markers, with no external benchmark or independent study. It is a definition, not an entity with falsifiable predictions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs." pith.science (2026). https://pith.science/paper/LKEVGUJH

@misc{pith2026241202602,
  author       = {Pith},
  title        = {Pith review of: CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKEVGUJH}},
  note         = {Machine review of arXiv:2412.02602}
}
read the original abstract

This paper analyzes the performance of Small Language Models (SLMs) and Vision Language Models (VLMs) and evaluates the trade-off between model performance and carbon emissions across 4 essential tasks: Image Captioning, Visual Question Answering (VQA), Dialogue Summarization and Text-to-SQL conversion. Various SLMs and VLMs belonging to the Qwen and LLaMA architecture family are chosen and variants based on model size in terms of the number of parameters, quantization level and fine-tuning parameters are evaluated. The model variant's performance and carbon emissions are calculated. To quantify the trade-off between model performance and carbon emissions, we introduce a novel metric called CEGI (Carbon Efficient Gain Index). This metric represents the carbon emission per unit percentage gain per million trainable parameters . This metric provides a normalized measure to compare model's efficiency in terms of performance improvement relative to their environmental cost. The experiment's outcome demonstrates that fine-tuning SLMs and VLMs can achieve performance levels comparable to Large Language Models (LLMs) while producing significantly less carbon emissions. Our findings suggest that the marginal gains in accuracy from larger models do not justify the substantial increase in carbon emissions. Leveraging lower-bit quantization levels, the proposed metric further enhances energy efficiency without compromising performance. This study highlights balancing high performance and environmental sustainability. It offers a valuable metric for selecting models suitable for environmentally-friendly AI development.

Figures

Figures reproduced from arXiv: 2412.02602 by the authors.

Figure 1
Figure 1. Performance comparison for image captioning [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 3
Figure 3. Comparison of ROUGE-1 scores for BM, FT , and GPT-4o in Dialogue Summarization. Fine-tuned mod￾els demonstrate substantial improvements over the base models and GPT-4o baseline, showcasing their efficacy in generating summaries with higher semantic relevance [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 5
Figure 5. Comparison of SPICE scores and carbon emissions for [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Comparison of BLEU scores and carbon emissions for [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison of ROUGE-1 scores and carbon emissions for [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Comparison of EA scores and carbon emissions for [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Carbon Emissions Across Model Tasks. Bars with the green notation indicate the carbon [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages

  1. [1]

    Energy and policy considerations for deep learning in nlp, 2019

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp, 2019

    2019

  2. [2]

    Carbon emissions and large neural network training, 2021

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training, 2021

    2021

  3. [3]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    2020

  5. [5]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  6. [6]

    Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F

    Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. Deep learning’s diminishing returns: The cost of improvement is becoming unsustainable. IEEE Spectrum, 58(10):50–55, 2021

    work page 2021

  7. [7]

    Towards the systematic reporting of the energy and carbon footprints of machine learning, 2022

    Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning, 2022

    work page 2022

  8. [8]

    Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Ra...

    work page 2022

Show all 32 references
  1. [9]

    Smith, Nicole DeCario, and Will Buchanan

    Jesse Dodge, Taylor Prewitt, Remi Tachet Des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexan- dra Sasha Luccioni, Noah A. Smith, Nicole DeCario, and Will Buchanan. Measuring the carbon intensity of ai in cloud instances, 2022

    2022

  2. [10]

    Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai, 2022

    Semen Budennyy, Vladimir Lazarev, Nikita Zakharenko, Alexey Korovin, Olga Plosskaya, Denis Dimitrov, Vladimir Arkhipkin, Ivan Oseledets, Ivan Barsola, Ilya Egorov, Aleksandra Kosterina, and Leonid Zhukov. Eco2ai: carbon emissions tracking of machine learning models as the firs...

    2022

  3. [11]

    S. A. Budennyy, V . D. Lazarev, N. N. Zakharenko, A. N. Korovin, O. A. Plosskaya, D. V . Dimitrov, V . S. Akhripkin, I. V . Pavlov, I. V . Oseledets, I. S. Barsola, I. V . Egorov, A. A. Kosterina, and L. E. Zhukov. eco2ai: Carbon emissions tracking of machine learning models a...

    2023

  4. [12]

    Measuring the intrinsic dimension of objective landscapes, 2018

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, 2018

    2018

  5. [13]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computatio...

    2021

  6. [14]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision – ECCV 2014, pages...

    2014

  7. [15]

    Medical visual question answering via conditional reasoning and contrastive learning

    Bo Liu, Li-Ming Zhan, Li Xu, and Xiao-Ming Wu. Medical visual question answering via conditional reasoning and contrastive learning. IEEE Transactions on Medical Imaging, 42(5):1532–1545, 2023

    2023

  8. [16]

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V . Jawahar, and Dimosthenis Karatzas. Scene text visual question answering, 2019

    2019

  9. [17]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural...

    2017

  10. [18]

    X-linear attention networks for image captioning

    Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10968–10977, 2020

    2020

  11. [19]

    Image captioning: Transforming objects into words

    Simão Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transforming objects into words. In Neural Information Processing Systems, 2019

    2019

  12. [20]

    Star-transformer

    Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star-transformer. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2019

  13. [21]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  14. [22]

    SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the 2nd Workshop on New Frontiers in Sum...

    2019

  15. [23]

    DialogSum: A real-life scenario dialogue summarization dataset

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dialogue summarization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 5062–5074, ...

    2021

  16. [24]

    MediaSum: A large-scale media interview dataset for dialogue summarization

    Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. MediaSum: A large-scale media interview dataset for dialogue summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yicha...

    2021

  17. [25]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Ellen R...

    2018

  18. [26]

    KaggleDBQA: Realistic evaluation of text-to- SQL parsers

    Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. KaggleDBQA: Realistic evaluation of text-to- SQL parsers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and...

    2021

  19. [27]

    The artbench dataset: Benchmarking generative models with artworks

    Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv:2206.11404, 2022

    2022 arXiv

  20. [28]

    Pathvqa: 30000+ questions for medical visual question answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  21. [29]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 17

    2002

  22. [30]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  23. [31]

    Synthetic-Text-To-SQL: A synthetic dataset for training language models to generate sql queries from natural language prompts, April 2024

    Yev Meyer, Marjan Emadi, Dhruv Nathawani, Lipika Ramaswamy, Kendrick Boyd, Maarten Van Segbroeck, Matthew Grossman, Piotr Mlocek, and Drew Newberry. Synthetic-Text-To-SQL: A synthetic dataset for training language models to generate sql queries from natural language prompts, A...

    2024

  24. [32]

    The Landscape with Dead Trees

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. CoRR, abs/1607.08822, 2016. 18 A Dataset A.1 artbench-pd-256x256 This dataset is a subset of public domain images from ArtBench, focusing on historical art...

    2016 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.