REVIEW 3 major objections 5 minor 34 references
Performance is not All You Need: Sustainability Considerations for Algorithms
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper proposes FMS and ASC, two metrics that combine a model's accuracy with its training energy consumption into a single sustainability score, and shows they re-rank models across multiple vision tasks.
desk verdict A well-meant but under-validated proposal for sustainability metrics; the ASC ranking depends on an unreported discretization parameter, so the main results are not reproducible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
FMS is defined as (1+beta^2)*P*E / (beta^2*P + E), with E(w)=e^{-alpha w}; the harmonic mean is the central device that punishes imbalance between accuracy and energy. ASC is the integral of the performance-energy curve, approximated by a Riemann sum over N intervals; this area acts as a composite indicator of how quickly and how efficiently a model reaches its final accuracy.
What would settle it
Run the same energy and accuracy traces through ASC with N = 10, N = 100, and N = 1000, and check whether the model ordering changes; if it does, the reported cross-task conclusions are artifacts of an arbitrary parameter. Alternatively, record two training runs that reach identical accuracy with identical total energy but different energy schedules (e.g., fast early improvement vs slow steady improvement) and test whether ASC distinguishes them despite identical cumulative curves.
Extended reading notes
Core claim
The paper introduces FMS and ASC as dimensionless sustainability metrics applicable across tasks. FMS maps energy consumption w to E(w)=e^{-alpha w} and combines it with any performance metric P via the harmonic mean, deliberately penalizing algorithms that are strong on one axis and weak on the other. ASC records performance against cumulative energy during training, normalizes energy to [0,1], and integrates the resulting sustainability curve by rectangular sampling; it rewards algorithms that achieve high accuracy early in training with little energy. The paper claims both metrics are scale-invariant with respect to energy magnitude, provided the decay parameter alpha or the energy normal
Load-bearing premise
The rankings produced by these metrics are assumed to be stable under the choice of the energy-decay parameter alpha and the discretization parameter N, but the paper does not state N for its main results and its own ablation shows ASC rankings reversing as N changes.
Editorial extensions
If this is right
- If the metrics are adopted, a model like SwinTransformer can rank above GcViT on ImageNet despite lower accuracy, because it uses far less energy for a modest accuracy loss.
- The metrics provide a quantitative target for model compression and neural architecture search, favoring models that achieve high accuracy early in training.
- The metrics extend beyond classification to any performance measure, so segmentation and pose-estimation benchmarks can be re-evaluated on sustainability.
- Because the metrics are claimed to be scale-invariant, the same scoring could be applied to small CIFAR-scale experiments and large ImageNet-scale ones.
Reading between the lines
- The authors leave the discretization parameter N unreported in the main tables; their own appendix shows ASC values and rankings changing sharply with N, so published ASC rankings should be read as conditional on an unspecified choice of N.
- The scale-invariance argument for FMS requires re-setting alpha whenever the energy scale changes; a truly parameter-free sustainability score would need a principled rule for alpha, not an ad hoc scaling.
- The same curve-and-area machinery could be applied to inference-phase energy, carbon intensity by region, or hardware-specific power draws, giving the framework a wider reach than the training-only benchmarks reported here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two sustainability metrics for deep learning: FMS (F-Measure on Sustainability), a harmonic mean of a performance metric P and an exponentially transformed energy measure E(w)=e^{-\alpha w}, and ASC (Area under Sustainability Curve), a Riemann-sum approximation of the area under a performance-versus-normalized-energy curve. The authors claim that both metrics are scale-invariant with respect to energy consumption and that experiments across image classification, segmentation, pose estimation, and batch/online learning demonstrate that they provide a reliable quantitative basis for cross-task algorithm comparison. The appendix reports ablations over N, wmax, and \beta, and compares against Score, SI, and SAM.
Significance. The motivation is timely and the paper addresses a real gap: most evaluation metrics ignore training energy. The proposed metrics are simple and the experimental coverage is broad, including several task families and a code release. If the metrics were shown to be robust and scale-invariant, they would be a useful addition to green-AI practice. However, as presented, the empirical validation is undermined by an unreported discretization parameter N that changes ASC values and even rankings, and the claimed scale invariance is achieved only by rescaling the free parameter \alpha. The core idea is salvageable, but the current claims are not supported by the evidence in the manuscript.
major comments (3)
- [Eq. (3), §4.1, Table 7 vs. Table 1] ASC depends on the discretization N, but the main experiments never state which N is used. Table 7 shows that ImageNet ASC values for EfficientNet/Swin/GcViT are 0.6658/0.7907/0.8010 at N=10, 0.6537/0.7851/0.7934 at N=50, 0.5783/0.7862/0.7858 at N=100, and 0.2082/0.2171/0.1724 at N=500. Table 1 reports 65.02/77.72/77.02, which matches none of these rows. Rankings also flip: GcViT is best at N=10, SwinTransformer is best at N=100, and GcViT becomes worst at N=500. Since N is unreported and the main-table values are not reproducible from the ablation, the central claim that ASC provides a reliable quantitative basis for algorithm comparison is unsupported.
- [§3.3, Eq. (1)] The claimed scale invariance of FMS is obtained by rescaling the free parameter \alpha: for a model consuming 1000 kWh, the authors set \alpha_2 = \alpha_1/1000, so E(w) is identical by construction. This is a normalization choice, not an intrinsic property of the metric; any metric with a free rate parameter can be made invariant in this way. Moreover, \alpha is chosen by an arbitrary rule in §4 ('100 times the energy consumption value at the 100th batch'), and its units are unclear. The conclusion's statement that 'both metrics are scale-invariant' is therefore not established.
- [§4.1, §4.3, Appendix B] The metric values and rankings depend on user-chosen parameters \alpha, wmax, and N. The text says training exits at 1 kWh for ASC, but Fig. 5 varies wmax and shows ASC increasing with it; no justification is given for the chosen value. The FMS \alpha choice changes by task and the ablation in Fig. 6 covers only one model pair. Since the main tables do not report these settings or a sensitivity analysis, the experiments do not demonstrate that FMS and ASC provide a parameter-free or universally comparable sustainability measure.
minor comments (5)
- [Throughout] There are repeated typos: 'FSM' appears instead of 'FMS' in Table 2, Fig. 3, and the text around them.
- [Table 8] The caption says 'pose recognition tasks' but the table reports segmentation models (MobileNet, PSANet, etc.); this appears to be a copy-paste error.
- [Eq. (3)] The notation is unclear: p_{iT/N} is used as a subscript and the sustainability function f is not explicitly defined. Please define f and the sampling indices precisely.
- [References] Reference [32] cites a genome-visualization paper for GcViT, but the model used in the experiments is a vision transformer; the citation is incorrect.
- [Experiments] No error bars, confidence intervals, or repeated runs are reported in any table; for an empirical metric comparison, this limits the strength of the conclusions.
Circularity Check
The claimed scale invariance of FMS and ASC is imposed by freely choosing alpha and wmax normalization, making the property true by construction rather than demonstrated; main-table ASC values also depend on an unreported discretization N.
-
self definitional
[Section 3.3 'Scale Invariance of Metrics on Electricity', Eq. (1)]
"For the FMS method, assuming that we have determined the parameter α1 when the algorithm’s energy consumption is 1 kWh, then if the large model consumes 1000 kWh on large data, then according to Eqn. (1), We take α2 = α1/1000 to maintain identical energy consumption metrics. Therefore, FMS can make energy consumption with different levels scale invariant."
The scale invariance asserted here is not a derived property of FMS; it is imposed by rescaling the free parameter α inversely with energy. Eq. (1), E(w)=e^{−αw}, and the instruction α2=α1/1000 make E(1000)=E(1) by construction. Since α is user-chosen and in the experiments is set per model/task (Sec. 4: 'α is set to 100 times the energy consumption (kWh) value at the 100th batch'), the cross-scale comparability is an input normalization, not an emergent characteristic. Claiming this as a property of the metric is therefore definitional.
-
self definitional
[Section 3.3, ASC normalization paragraph; conclusion]
"It is worth noting that ASC needs to normalize the power value when calculating the area under the sustainability curve. Assuming that for algorithm A and algorithm B, wmax=1kWh and wmax=1000kWh, then the point wi=0.8 kWh on algorithm A will correspond to the point wj=800kWh on algorithm B, because these two points have the same weight."
The ASC 'scale invariance' is obtained by dividing all energy values by wmax, but the metric value itself depends on wmax: Fig. 5 shows ASC increasing with wmax for all three models, and Table 7 shows ASC changing and rankings reversing with the discretization N. Thus the conclusion that 'Both metrics are scale-invariant with respect to energy consumption' is not a property of the definition; it is a normalization choice that the paper then treats as a demonstrated result.
full rationale
The core definitions of FMS and ASC are new and are evaluated on external benchmarks, so the paper is not wholly circular. However, the paper's key advertised property—scale invariance—is manufactured by free choices of α and wmax normalization, making the claim true by construction rather than by empirical demonstration. In addition, the main-table ASC values depend on an unreported discretization parameter N: Appendix B/Table 7 shows EfficientNet's ASC on ImageNet dropping from 0.5783 (N=100) to 0.2082 (N=500) and rankings flipping among models, yet the main tables report ASC values without specifying N, so the headline 'quantitative basis' is not reproducible from the paper's stated parameters. The SAM baseline [11] is authored by overlapping authors (Gowda, Jin), but it is used only as a comparison criterion, not as justification for the derivation, so it does not add circularity. The scale-invariance steps are the load-bearing circular elements, giving a partial circularity score of 6.
Assumptions & free parameters
free parameters (4)
- alpha in E(w)=e^{-alpha w} =
100 times energy at 100th batch (classification, online/batch) or at 1000th iteration (segmentation, pose)
- beta in FMS =
1 (default), 0.5, 2 in ablation
- wmax cutoff in ASC =
1 kWh in experiments; varied 0.8-1.8 in ablation
- N intervals in ASC =
Not stated in main experiments; 10, 50, 100, 500 in ablation
assumptions (3)
- ad hoc to paper Exponential decay E(w)=e^{-alpha w} is an appropriate normalization of training energy into [0,1].
- domain assumption Energy consumption can be measured accurately and reproducibly during training.
- domain assumption The performance-energy curve is smooth enough for rectangular integration to approximate the area.
invented entities (2)
-
FMS (F-Measure on Sustainability)
-
ASC (Area under Sustainability Curve)
Cite this review
Pith. "Pith review of Performance is not All You Need: Sustainability Considerations for Algorithms." pith.science (2026). https://pith.science/paper/FNUWIGI2
@misc{pith2026250900045,
author = {Pith},
title = {Pith review of: Performance is not All You Need: Sustainability Considerations for Algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/FNUWIGI2}},
note = {Machine review of arXiv:2509.00045}
}
read the original abstract
This work focuses on the high carbon emissions generated by deep learning model training, specifically addressing the core challenge of balancing algorithm performance and energy consumption. It proposes an innovative two-dimensional sustainability evaluation system. Different from the traditional single performance-oriented evaluation paradigm, this study pioneered two quantitative indicators that integrate energy efficiency ratio and accuracy: the sustainable harmonic mean (FMS) integrates accumulated energy consumption and performance parameters through the harmonic mean to reveal the algorithm performance under unit energy consumption; the area under the sustainability curve (ASC) constructs a performance-power consumption curve to characterize the energy efficiency characteristics of the algorithm throughout the cycle. To verify the universality of the indicator system, the study constructed benchmarks in various multimodal tasks, including image classification, segmentation, pose estimation, and batch and online learning. Experiments demonstrate that the system can provide a quantitative basis for evaluating cross-task algorithms and promote the transition of green AI research from theory to practice. Our sustainability evaluation framework code can be found here, providing methodological support for the industry to establish algorithm energy efficiency standards.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[11]
arXiv preprint arXiv:2310.06522 (2023)
Gowda, S.N., Hao, X., Li, G., Gowda, S.N., Jin, X., Sevilla-Lara, L.: Watt for what: Rethinking deep learning’s energy-performance relationship. arXiv preprint arXiv:2310.06522 (2023)
arXiv 2023
-
[1]
Alyamkin, S., Ardi, M., Berg, A.C., Brighton, A., Chen, B., Chen, Y., Cheng, H.P., Fan, Z., Feng, C., Fu, B., et al.: Low-power computer vision: Status, challenges, and opportunities. IEEE J. Emerg. Sel. Top. Circuits Sys9(2), 411–421 (2019)
work page 2019
-
[2]
arXiv preprint arXiv:2007.03051 (2021)
Anthony, L., Kanding, B., Selvan, R.: Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051 (2021)
arXiv 2007
-
[3]
Springer-Verlag, Berlin, Heidelberg (2006)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer-Verlag, Berlin, Heidelberg (2006)
work page 2006
-
[4]
Chen, L., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. CoRR (2017)
work page 2017
-
[5]
https://github.com/open-mmlab/mmpose (2020)
Contributors, M.: MMPose: Openmmlab pose estimation toolbox and benchmark. https://github.com/open-mmlab/mmpose (2020)
work page 2020
-
[6]
In: CVPR (2016)
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
2016
-
[7]
In: 2009 IEEE Conference on Computer Vision and Pattern Recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009).https://doi.org/10.1109/CVPR.2009. 5206848
Show all 34 references
-
[8]
arXiv preprint arXiv:2010.11929 (2020)
Dosovitskiy Alexey, e.a.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
2010 arXiv
-
[9]
arXiv preprint arXiv:2309.14393 (2023)
Faiz,A.,Kaneda,S.,Wang,R.,Osi,R.,Sharma,P.,Chen,F.,Jiang,L.:Llmcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393 (2023)
2023 arXiv
-
[10]
Sustainability 15(18) (2023)
Fan, Z., Yan, Z., Wen, S.: Deep learning and artificial intelligence in sustainabil- ity: A review of sdgs, renewable energy, and environmental health. Sustainability 15(18) (2023)
2023
-
[12]
arXiv preprint cs/0207029 (2002)
Graves, A., Schmidhuber, J.: Bidirectional lstm networks for handwriting recogni- tion. arXiv preprint cs/0207029 (2002)
2002 arXiv
-
[13]
In: ICCV
He, J., Deng, Z., Zhou, L., Wang, Y., Qiao, Y.: Adaptive pyramid context network for semantic segmentation. In: ICCV. pp. 7519–7528 (2019)
2019
-
[14]
In: CVPR
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016) Sustainability Considerations for Algorithms 15
2016
-
[15]
https://github.com/mlco2/codecarbon (2021), accessed: 2024-11- 23
Henderson, P., Hu, J., Jong, W.S., Peyton, J., Prabhu, V., Rahtu, E., Xiao, R.: Codecarbon. https://github.com/mlco2/codecarbon (2021), accessed: 2024-11- 23
2021
-
[16]
Information Sciences453, 50–65 (2018)
Jin, X.B., Geng, G.G., Xie, G.S., Huang, K.: Approximately optimizing ndcg using pair-wise loss. Information Sciences453, 50–65 (2018)
2018
-
[17]
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. . (2009)
2009
-
[18]
In: ICML (2019)
Lacoste, A., Diaby, A., Mielke, A., Parot, J., Chereau, G., Litz, P., Ricci, F.: Quantifying the carbon emissions of machine learning. In: ICML (2019)
2019
-
[19]
In: Proceedings of the 1st International Workshop on Green and Sustainable Software
Leclerc, G., Stévenin, V., Perache, M., Rouvoy, R., Merle, P., Seinturier, L.: Pow- erapi: A software library to monitor the energy consumed at the process level. In: Proceedings of the 1st International Workshop on Green and Sustainable Software. pp. 1–7 (2015)
2015
-
[20]
In: ICCV
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV. pp. 10012–10022 (2021)
2021
-
[21]
CSUR (2018)
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M.P., Shyu, M.L., Chen, S.C., Iyengar, S.S.: A survey on deep learning: Algorithms, techniques, and applications. CSUR (2018)
2018
-
[22]
Kaggle (2015)
Pozzolo, A.D.: Credit card fraud detection. Kaggle (2015)
2015
-
[23]
Scientific Reports14 (2024)
Rainio, O., Teuho, J., Klén, R.: Evaluation metrics and statistical tests for machine learning. Scientific Reports14 (2024)
2024
-
[24]
Repository, U.M.L.: Image segments dataset.https://archive.ics.uci.edu/ml/ datasets/Image+Segmentation (1990)
1990
-
[25]
Communications of the ACM (2020)
Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green ai. Communications of the ACM (2020)
2020
-
[26]
Foundations and Trends® in Machine Learning (2011)
Shalev-Shwartz, S.: Online learning and online convex optimization. Foundations and Trends® in Machine Learning (2011)
2011
-
[27]
In: UEMCON
Sinha, D., El-Sharkawy, M.: Thin mobilenet: An enhanced mobilenet architecture. In: UEMCON. pp. 0280–0285. IEEE (2019)
2019
-
[28]
In: AAAI
Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: AAAI. vol. 33, pp. 10552–10560 (2019)
2019
-
[29]
NeurIPS (2017)
Vaswani, A.: Attention is all you need. NeurIPS (2017)
2017
-
[30]
In: NeurIPS
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS. vol. 30 (2017)
2017
-
[31]
TPAMI43(10), 3349–3364 (2020)
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. TPAMI43(10), 3349–3364 (2020)
2020
-
[32]
BMC genomics (2020)
Wilkey, A.P., Brown, A.V., Cannon, S.B., Cannon, E.K.: Gcvit: a method for interactive, genome-wide visualization of resequencing and snp array data. BMC genomics (2020)
2020
-
[33]
arXiv preprint arXiv:2108.12617 (2021)
Yu, H., Xu, Y., Zhang, J., Zhao, W., Guan, Z., Tao, D.: Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617 (2021)
2021 arXiv
-
[34]
In: ECCV (2018) 16 X
Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C.C., Lin, D., Jia, J.: Psanet: Point-wise spatial attention network for scene parsing. In: ECCV (2018) 16 X. Li, C. Zhang et al. Appendix A Details on Tasks A.1 Image Classification The transform pipeline standardizes the raw images...
2018
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.