{"id":"dfe425fe-c04c-44ac-9df2-aa1f10780d82","arxiv_id":"2412.02602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using their new CEGI metric, the authors report that fine-tuned small models and low-bit quantization deliver near-large-model performance at a fraction of the carbon cost.","lead":"This paper measures how much carbon is released when fine-tuning small language models and vision-language models, and introduces a score called CEGI that compares performance gain to emissions. The authors find that smaller, fine-tuned models can match larger models' accuracy while emitting far less carbon, so bigger is not always better for the environment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CEGI's base-relative percentage gain (Eqs. 4–10) rewards low base scores and is not comparable across metrics; the model-selection and trade-off claims rest on this unvalidated normalization.","rationale":"The central contribution is CEGI, and the conclusion that practitioners should prefer certain small/quantized models is derived from CEGI rankings (Tables 10–13). The weakest link is therefore the normalization in Eqs. 4–10. The reader's weakest assumption identifies exactly this: percentage gain over a model's own base is scale-dependent and not comparable across metrics. My analysis confirms and sharpens it: Table 7's base BLEU values of 0.001–0.0037 turn tiny absolute improvements into thousands of percent, making CEGI rank Qwen-VL-7B best partly because its base score is lowest. A common-reference or absolute-threshold recomputation would be a decisive check. I also note internal inconsistencies (the trainable-parameter column in Tables 10–13 does not match the sum notation of Eq. 10/Table 20, and reproduced CEGI values differ), which further weaken the reliability claim. None of this attacks the empirical direction; the measured emissions and performance improvements for these specific tasks could be real, and the paper explicitly limits scope in Section 7. But the headline claims about the metric and the marginal-gain conclusion should not be accepted until the normalization is validated or replaced. Since the reader already made the verdict conditional on exactly this, I leave the verdict unchanged.","tokens_in":22167,"tokens_out":13994,"duration_ms":128545,"concrete_test":"Recompute the CEGI rankings in Tables 10–13 two ways: (1) using gain relative to a common reference (GPT-4o zero-shot, where available) instead of each model's own base, with a minimum absolute performance threshold (e.g., BLEU > 0.05, EA > 0.75, ROUGE-1 > 0.40) to exclude near-zero baselines; and (2) using the per-configuration definition CEGI_config = CE_config / (Gain_config × Tp_config) with Table 20 Tp values, then averaging. Check whether the argmin-CEGI model per task stays the same and whether the Table 10–13 numbers can be reproduced. If the optimal model changes or the numbers do not reproduce, the metric is not a reliable basis for the paper's model-selection and trade-off conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is CEGI, but the normalization in Eqs. 4–10 makes rankings depend on percentage gain relative to each model's own base score, which is not comparable across metrics or tasks. In Visual-QA, base BLEU scores are 0.0010–0.0037 (Table 7), so fine-tuning produces gains of 1,500–7,570% and near-zero CEGIs (Table 11); if the base BLEU of Qwen-VL-7B were 0.002 instead of 0.001, the reported ~7,570% gain would fall to ~3,785%, halving the apparent efficiency. The metric thus rewards low baselines rather than absolute performance. Although Eq. 4 also defines gain vs GPT-4o, Eq. 5 and Tables 10–13 use only base-relative gain, so models below GPT-4o (e.g., Llama-3.2-3B in Text-to-SQL, EA gain vs GPT-4o = -3.5%) can still get favorable CEGI values. A 10% SPICE gain and a 10% BLEU gain are not equivalent units, yet CEGI aggregates percentage gains over Qb/Lr configurations and ranks models within and across tasks (Tables 10–13). In addition, the CEGI values in Tables 10–13 are not reproducible from Eq. 10 and Table 20 under a straightforward reading: the 'Train params' column appears to be an average over LoRA ranks rather than the sum the notation requires, and the |Lr| factor in Eq. 9 changes the normalization. Since the conclusion that CEGI 'identifies the same model selected through human judgment' (Section 6.3) and the practical recommendations both rest on these rankings, this scale dependence is load-bearing. Section 7's limitation statement narrows the scope to specific datasets and Qwen/Llama families but does not address the metric's validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CEGI (Carbon Efficient Gain Index), a metric that combines carbon emissions, percentage performance gain over a base model, and trainable-parameter count, to rank fine-tuned small language models (SLMs) and vision-language models (VLMs) against larger models across four tasks: image captioning, visual question answering, dialogue summarization, and text-to-SQL. The authors fine-tune Qwen and LLaMA family models with LoRA at ranks 4, 8, 16, and 32 and at 4-bit and 8-bit quantization, measure carbon footprints with eco2AI, and report performance and emissions for each configuration. They conclude that fine-tuned SLMs and VLMs can achieve performance comparable to larger models at substantially lower carbon cost, and that CEGI identifies the optimal model in each task consistently with human judgment.","tokens_in":22600,"tokens_out":7642,"duration_ms":65671,"significance":"If the CEGI metric were valid, it would provide a practically useful tool for carbon-aware model selection. The empirical direction—showing that small fine-tuned models can reach near-large-model performance at lower emissions—is timely and relevant for the Green AI community. The paper covers a diverse set of tasks and model families, and the use of eco2AI for emissions tracking is a positive step toward reproducible carbon accounting. The main contribution, however, is the metric itself, and the construction and validation of that metric currently have serious problems that undermine the paper's central claims.","major_comments":[{"comment":"The base-relative percentage gain of Eq. (4) is scale-dependent and not comparable across metrics. In Visual-QA, Table 7 shows base BLEU values of 0.0010-0.0037, so fine-tuning yields percentage gains of 1,500-7,570% and near-zero CEGI values in Table 11. Changing the base BLEU of Qwen-VL-7B from 0.0010 to 0.0020 would reduce the reported 7,570% gain to about 3,785%, roughly doubling the CEGI. This means the metric rewards low baseline scores rather than absolute performance improvement, and the CEGI rankings that support the model-selection conclusions are not robust to small changes in the input data.","section":"Section 6.3, Eq. (4)-(10), Tables 7 and 11"},{"comment":"The CEGI values reported in Tables 10-13 are not reproducible from the stated formulas and the data provided. For image captioning, Qwen-VL-7B has total CE=137.97g, average gain=166.67%, and trainable parameters from Table 20 (averaged over Lr=8,16,32 gives 3.12m; summed gives 9.37m); substituting these into Eq. (10) yields CEGI≈0.265, but Table 10 reports 0.36. Similar discrepancies appear for Qwen-VL-2B in Table 10 and Qwen-VL-7B in Table 11. Additionally, the 'Train params' column in Table 12 for Qwen2.5-0.5B is 0.36m, which matches neither the sum (1.26m) nor the average (0.42m) of the Table 20 values for Lr=4,8,16 used in the dialogue summarization experiments. The rankings and the practical recommendations built on them cannot be verified without a correct and consistent computation.","section":"Section 6.3, Eqs. (9)-(10), Tables 10-13 and Table 20"},{"comment":"The claim that the lowest-CEGI model 'consistently identifies the same model selected through human judgment' is unsupported. No description of the human-selection procedure, criteria, or judgment data is provided, so this validation is circular: CEGI is computed from the same performance and emissions measurements that would also inform any human selection, and there is no independent evidence that CEGI matches human judgment. The authors need to specify the human-selection protocol and provide the resulting choices for each task.","section":"Section 6.3, final paragraph"},{"comment":"The paper states that every reported value, including performance metrics and emissions, is the mean of 5 independent runs, but no variance, standard deviation, or confidence interval is reported anywhere. Given that the CEGI differences between models are sometimes very small (e.g., 0.005 vs 0.034 in Table 11) and that ROUGE-1 scores are often tied at 0.50-0.51, the reader cannot determine whether the claimed trade-offs and rankings are statistically meaningful. Error bars or per-run values should be provided for at least the key tables.","section":"Section 5, opening paragraph"},{"comment":"The GPT-4o baseline is not a matched comparison: it is evaluated zero-shot, with no fine-tuning, no carbon-emission measurement, and a different prompt setup. The claim that 'fine-tuned models surpass GPT-4o' (Section 6.1.1) therefore compares adapted small models to an unadapted generalist, not to an equivalently fine-tuned LLM. This weakens the abstract's broader conclusion that fine-tuned SLMs and VLMs achieve performance 'comparable to LLMs,' since the comparison does not isolate model size as the variable of interest.","section":"Sections 5.1, 5.2, and 6.1"}],"minor_comments":[{"comment":"The metric is called 'Carbon Efficient Gain Index' in the abstract and Section 6.3, but 'Carbon Efficient Emission Index' in Section 7; the terminology should be made consistent throughout.","section":"Abstract vs. Section 7"},{"comment":"The row label 'Test-to-SQL' is a typo for 'Text-to-SQL', and the Visual QA metric is written as 'BLUE' instead of 'BLEU'.","section":"Table 1"},{"comment":"The phrase 'with lower emmisions compared to Qwen2.5-14B' contains a spelling error ('emmisions' should be 'emissions').","section":"Section 6.2"},{"comment":"The sentence 'the percentage improvement is more significant in smaller models (e.g., Qwen-VL-7B's 171.43% vs. Llama-3.2-11B's 72.22%)' contrasts models that differ in architecture and family, not just size; a within-family comparison (e.g., Qwen-VL-2B vs Qwen-VL-7B) would be clearer.","section":"Section 6.1.1"},{"comment":"The limitation statement reads 'our focus on Qwen and Llama variants may represent the broader landscape of model architectures,' but the intended meaning appears to be 'may not represent'; the missing 'not' changes the sentence into the opposite of what is likely meant.","section":"Section 7, Limitations"},{"comment":"Table 3 lists a GPT-4o BLEU value of 0.0013 for Qwen-VL-2B, but Table 7 computes gains versus GPT-4o using GPT-4o BLEU = 0.0037; the inconsistency should be resolved.","section":"Table 3 and Table 7"},{"comment":"The Visual-QA performance plots use an exponential or log scale for BLEU scores, as indicated by the 'e3x scale' label in Figure 2, but the captions do not explain that the apparent visual gaps correspond to very different absolute differences; this should be stated explicitly.","section":"Figures 2 and 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real problem and the empirical trend is plausible, but the central metric is currently not reproducible and its normalization is scale-dependent. I would encourage the editor to ask for the code and raw data tables as part of the revision, since independent verification of the CEGI values is essential. The GPT-4o comparison should be reframed as a zero-shot reference rather than evidence about model-size trade-offs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the empirical story -- smaller fine-tuned models get close to larger ones at a fraction of the carbon cost -- is plausible from the raw numbers, but the CEGI metric as presented is not usable. The values in Tables 10-13 do not reproduce from Equations 4-10 on a straightforward reading, and the normalization rewards low baselines in a way that dominates the rankings.\n\nWhat's genuinely new: the paper reports measured carbon emissions for LoRA fine-tuning (multiple ranks, 4/8-bit) of Qwen and Llama variants on four tasks using eco2AI. Those tables are the useful part. I don't know of another paper with this exact configuration sweep. The qualitative finding about smaller models is not new, but the specific measurements are a data point others can build on.\n\nThe soft spots are serious and centered on the metric. First, Eq. (10) defines the denominator with Σ over LoRA ranks of Tp, but the 'Train params' column in Table 10 (1.67m for Qwen-VL-2B, 3.12m for Qwen-VL-7B) is the mean of Tp over ranks 8, 16, 32 from Table 20, not the sum. I tried to reproduce CEGI 0.65 and couldn't. Second, the base-relative percentage gain in Eq. (4) is not comparable across tasks: BLEU scores near 0.001 in Visual QA produce gains of 1,500-7,570%, so the low-CEGI winners are largely an artifact of low baselines. Third, the 'human judgment' validation in Section 6.3 is never described, so the claim that CEGI matches it is unfalsifiable.\n\nMinor: no error bars are reported despite 5-run means; the GPT-4o baseline is zero-shot and proprietary, so the comparisons are not matched; Table 7 has a clear typo (5,538% vs 1,972% for the same model at neighboring ranks). The limitations section is honest about scope but doesn't acknowledge the metric's scale dependence.\n\nBottom line: the raw emissions table is a contribution, but the central index needs a full rewrite and external validation before the trade-off conclusions can be trusted. I'd send it to review -- a good referee can force the repair -- but I would not cite it in its current form.","headline":"The raw emissions measurements are a useful data point, but CEGI does not reproduce from its own equations and its percentage-gain normalization rewards low baselines, so the central metric needs a major rework before the trade-off conclusions can be trusted.","tokens_in":23129,"tokens_out":5602,"would_cite":false,"duration_ms":54985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned small models rival much larger LLMs while emitting far less carbon, and CEGI quantifies the trade-off.","keywords":["carbon-efficient gain index","small language models","vision language models","low-rank adaptation","quantization","carbon emissions","sustainable AI","model selection"],"falsifier":"Recompute CEGI for the paper's own tables using absolute score differences rather than percentage gains, or add a small floor to near-zero baselines such as the 0.0037 BLEU score; if the lowest-CEGI model changes for any of the four tasks, the ranking is an artifact of percentage scaling rather than a stable measure of efficiency.","tokens_in":21964,"feed_emoji":"🌱","tokens_out":7648,"duration_ms":70984,"temperature":0.7,"pith_summary":"The paper sets out to show that for image captioning, visual question answering, dialogue summarization, and text-to-SQL, small language models and vision-language models fine-tuned with LoRA can reach scores close to those of much larger models while emitting far less carbon. To make this comparison quantitative, the authors define CEGI, the carbon emission per unit percentage gain per million trainable parameters. Across their experiments, smaller fine-tuned variants such as Qwen2.5-0.5B and Qwen-VL-2B achieve most of the accuracy of the largest tested models, and the metric identifies the same sweet-spot models that a human would choose from the plots. The paper concludes that the marginal accuracy gained from larger models is not worth the extra emissions, and that 4-bit and 8-bit quantization lower energy use without hurting scores. A sympathetic reader would take this as an argument to prefer small, task-tuned, quantized models when both accuracy and environmental cost matter.","feed_headline":"Small tuned models match big LLMs at a fraction of the carbon","feed_subtitle":"A new Carbon Efficient Gain Index shows the smallest Qwen and LLaMA models deliver most of the accuracy for a fraction of the CO2.","key_machinery":"The load-bearing object is the CEGI score, defined as $G^{o}_{M,\\mu,T_p} = \\frac{(\\sum CE) \\times |L_r|}{(\\sum G_{M,\\mu}(FT,BM)) \\times (\\sum T_p)}$, giving grams of CO2 per percentage point of gain per million trainable parameters. The other two mechanisms feed into it: LoRA keeps the trainable-parameter denominator small and task-specific, and 4-bit and 8-bit quantization cut the energy use that appears in the numerator while leaving scores nearly unchanged. The identity works because a lower CEGI means a model buys more performance per gram of carbon per parameter.","core_discovery":"The authors claim that fine-tuned SLMs and VLMs deliver performance comparable to LLMs at substantially lower carbon cost, and that CEGI orders models by that trade-off. For example, in dialogue summarization Qwen2.5-0.5B reaches a ROUGE-1 of 0.42 versus 0.51 for Qwen2.5-14B while emitting about 25 g instead of about 154 g of CO2; on image captioning Qwen-VL-2B reaches 0.35 SPICE versus 0.31 for Llama-3.2-11B while emitting 97 g instead of 222 g. The lowest-CEGI model is Qwen-VL-7B for both image captioning and visual QA, Llama-3.2-1B for dialogue summarization, and Qwen2.5-3B for text-to-SQL, and these are the same models the authors mark as the performance-emissions sweet spots. On this evidence, the paper asserts that for these tasks larger models' incremental accuracy does not justify their disproportionate emissions.","pith_inferences":["As an editorial extension, CEGI's percentage-gain normalization is risky for metrics with near-zero baselines, so a robust version of the index might use absolute gains, a floor, or a saturating transformation before it is used as a general benchmark.","The paper measures emissions mainly from fine-tuning on one GPU; extending the same index to inference, deployment, and differing hardware would test whether the ranking persists outside the lab.","A testable extension would be to recompute CEGI on a task where the base model is already strong; if rankings invert, the index is rewarding models that start weak rather than models that are genuinely efficient."],"forward_implications":["A practitioner working on these four tasks can pick a fine-tuned 0.5B or 2B model and get most of the accuracy of a 7B, 11B, or 14B model while cutting measured carbon emissions by more than half.","Lower-bit quantization at 4-bit and 8-bit becomes a safe default: performance varies by less than a couple of percent while emissions drop, so efficiency gains do not have to be traded against accuracy.","CEGI gives a single number for ranking candidate models on performance-per-carbon, which could be adopted in model-selection workflows whenever emission measurements are available.","The reported pattern implies that model families should be evaluated at their small end before scaling up, because the largest variants' accuracy gains over the small variants are small relative to their emissions."],"supporting_citations":[{"why":"Baseline evidence that large-model training emits substantial CO2, motivating the study.","marker":"[1]"},{"why":"Quantification of carbon emissions from large neural network training, the problem the paper addresses.","marker":"[2]"},{"why":"The LoRA method that restricts trainable parameters in every fine-tuning run.","marker":"[3]"},{"why":"Carbon-emission tracking library used for all measurements.","marker":"[10]"},{"why":"Dialogue summarization dataset used for the SAMSum task.","marker":"[22]"},{"why":"Pathology VQA dataset used for the visual question answering task.","marker":"[28]"},{"why":"Artbench image-captioning subset used for the captioning task.","marker":"[27]"},{"why":"Synthetic text-to-SQL dataset used for the SQL generation task.","marker":"[31]"}],"fun_headline_variants":["Small tuned models match big LLMs, but with far less carbon","CEGI metric shows small models hit big accuracy with low CO2","Why the smallest tuned models are the carbon-efficient pick","New metric: small models match LLM performance per gram of CO2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"CEGI assumes that a percentage gain over the base model is a fair, comparable measure across metrics with very different scales and saturation points, and that dividing total emissions by that percentage and by trainable parameters yields a meaningful efficiency ranking.","fun_headline_variants_meta":{"raw":{"variants":["Small tuned models match big LLMs, but with far less carbon","CEGI metric shows small models hit big accuracy with low CO2","Why the smallest tuned models are the carbon-efficient pick","New metric: small models match LLM performance per gram of CO2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000378,"raw_usage":{"total_tokens":2053,"prompt_tokens":1032,"completion_tokens":1021,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":948}},"tokens_in":648,"tokens_out":1021,"duration_ms":9864,"temperature":1.0,"reasoning_tokens":948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:16:21.800960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute CEGI for the paper's own tables using absolute score differences rather than percentage gains, or add a small floor to near-zero baselines such as the 0.0037 BLEU score; if the lowest-CEGI model changes for any of the four tasks, the ranking is an artifact of percentage scaling rather than a stable measure of efficiency.","supporting_citations":[{"cited_title":"Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai, 2022","cited_arxiv_id":null,"evidence_quote":"Carbon-emission tracking library used for all measurements."},{"cited_title":"SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization","cited_arxiv_id":null,"evidence_quote":"Dialogue summarization dataset used for the SAMSum task."},{"cited_title":"Synthetic-Text-To-SQL: A synthetic dataset for training language models to generate sql queries from natural language prompts, April 2024","cited_arxiv_id":null,"evidence_quote":"Synthetic text-to-SQL dataset used for the SQL generation task."}],"review_version":1}