Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Performance is not All You Need: Sustainability Considerations for Algorithms

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read This paper proposes FMS and ASC, two metrics that combine a model's accuracy with its training energy consumption into a single sustainability score, and shows they re-rank models across multiple vision tasks.

desk verdict A well-meant but under-validated proposal for sustainability metrics; the ASC ranking depends on an unreported discretization parameter, so the main results are not reproducible. read the letter →

arxiv 2509.00045 v2 pith:FNUWIGI2 submitted 2025-08-24 cs.CV cs.PF

classification cs.CVcs.PF MSC 68T0768T45
keywords sustainabilitymetricsenergyefficiencygreenAIharmonicmeanareaundercurveperformance-energytrade-offFMSASC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that judging deep learning algorithms solely by accuracy ignores their environmental cost, and proposes two metrics that fold energy consumption into the score. FMS takes the harmonic mean of accuracy and an exponentially decaying energy factor, while ASC measures the area under a curve of accuracy versus energy used during training. The authors demonstrate these scores on image classification, segmentation, pose estimation, and online and batch learning, and contend they give developers a quantitative way to choose algorithms that balance performance and energy use.

What carries the argument

FMS is defined as (1+beta^2)*P*E / (beta^2*P + E), with E(w)=e^{-alpha w}; the harmonic mean is the central device that punishes imbalance between accuracy and energy. ASC is the integral of the performance-energy curve, approximated by a Riemann sum over N intervals; this area acts as a composite indicator of how quickly and how efficiently a model reaches its final accuracy.

What would settle it

Run the same energy and accuracy traces through ASC with N = 10, N = 100, and N = 1000, and check whether the model ordering changes; if it does, the reported cross-task conclusions are artifacts of an arbitrary parameter. Alternatively, record two training runs that reach identical accuracy with identical total energy but different energy schedules (e.g., fast early improvement vs slow steady improvement) and test whether ASC distinguishes them despite identical cumulative curves.

Watch

Extended reading notes

Core claim

The paper introduces FMS and ASC as dimensionless sustainability metrics applicable across tasks. FMS maps energy consumption w to E(w)=e^{-alpha w} and combines it with any performance metric P via the harmonic mean, deliberately penalizing algorithms that are strong on one axis and weak on the other. ASC records performance against cumulative energy during training, normalizes energy to [0,1], and integrates the resulting sustainability curve by rectangular sampling; it rewards algorithms that achieve high accuracy early in training with little energy. The paper claims both metrics are scale-invariant with respect to energy magnitude, provided the decay parameter alpha or the energy normal

Load-bearing premise

The rankings produced by these metrics are assumed to be stable under the choice of the energy-decay parameter alpha and the discretization parameter N, but the paper does not state N for its main results and its own ablation shows ASC rankings reversing as N changes.

Editorial extensions

If this is right

  • If the metrics are adopted, a model like SwinTransformer can rank above GcViT on ImageNet despite lower accuracy, because it uses far less energy for a modest accuracy loss.
  • The metrics provide a quantitative target for model compression and neural architecture search, favoring models that achieve high accuracy early in training.
  • The metrics extend beyond classification to any performance measure, so segmentation and pose-estimation benchmarks can be re-evaluated on sustainability.
  • Because the metrics are claimed to be scale-invariant, the same scoring could be applied to small CIFAR-scale experiments and large ImageNet-scale ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave the discretization parameter N unreported in the main tables; their own appendix shows ASC values and rankings changing sharply with N, so published ASC rankings should be read as conditional on an unspecified choice of N.
  • The scale-invariance argument for FMS requires re-setting alpha whenever the energy scale changes; a truly parameter-free sustainability score would need a principled rule for alpha, not an ad hoc scaling.
  • The same curve-and-area machinery could be applied to inference-phase energy, carbon intensity by region, or hardware-specific power draws, giving the framework a wider reach than the training-only benchmarks reported here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two sustainability metrics for deep learning: FMS (F-Measure on Sustainability), a harmonic mean of a performance metric P and an exponentially transformed energy measure E(w)=e^{-\alpha w}, and ASC (Area under Sustainability Curve), a Riemann-sum approximation of the area under a performance-versus-normalized-energy curve. The authors claim that both metrics are scale-invariant with respect to energy consumption and that experiments across image classification, segmentation, pose estimation, and batch/online learning demonstrate that they provide a reliable quantitative basis for cross-task algorithm comparison. The appendix reports ablations over N, wmax, and \beta, and compares against Score, SI, and SAM.

Significance. The motivation is timely and the paper addresses a real gap: most evaluation metrics ignore training energy. The proposed metrics are simple and the experimental coverage is broad, including several task families and a code release. If the metrics were shown to be robust and scale-invariant, they would be a useful addition to green-AI practice. However, as presented, the empirical validation is undermined by an unreported discretization parameter N that changes ASC values and even rankings, and the claimed scale invariance is achieved only by rescaling the free parameter \alpha. The core idea is salvageable, but the current claims are not supported by the evidence in the manuscript.

major comments (3)
  1. [Eq. (3), §4.1, Table 7 vs. Table 1] ASC depends on the discretization N, but the main experiments never state which N is used. Table 7 shows that ImageNet ASC values for EfficientNet/Swin/GcViT are 0.6658/0.7907/0.8010 at N=10, 0.6537/0.7851/0.7934 at N=50, 0.5783/0.7862/0.7858 at N=100, and 0.2082/0.2171/0.1724 at N=500. Table 1 reports 65.02/77.72/77.02, which matches none of these rows. Rankings also flip: GcViT is best at N=10, SwinTransformer is best at N=100, and GcViT becomes worst at N=500. Since N is unreported and the main-table values are not reproducible from the ablation, the central claim that ASC provides a reliable quantitative basis for algorithm comparison is unsupported.
  2. [§3.3, Eq. (1)] The claimed scale invariance of FMS is obtained by rescaling the free parameter \alpha: for a model consuming 1000 kWh, the authors set \alpha_2 = \alpha_1/1000, so E(w) is identical by construction. This is a normalization choice, not an intrinsic property of the metric; any metric with a free rate parameter can be made invariant in this way. Moreover, \alpha is chosen by an arbitrary rule in §4 ('100 times the energy consumption value at the 100th batch'), and its units are unclear. The conclusion's statement that 'both metrics are scale-invariant' is therefore not established.
  3. [§4.1, §4.3, Appendix B] The metric values and rankings depend on user-chosen parameters \alpha, wmax, and N. The text says training exits at 1 kWh for ASC, but Fig. 5 varies wmax and shows ASC increasing with it; no justification is given for the chosen value. The FMS \alpha choice changes by task and the ablation in Fig. 6 covers only one model pair. Since the main tables do not report these settings or a sensitivity analysis, the experiments do not demonstrate that FMS and ASC provide a parameter-free or universally comparable sustainability measure.
minor comments (5)
  1. [Throughout] There are repeated typos: 'FSM' appears instead of 'FMS' in Table 2, Fig. 3, and the text around them.
  2. [Table 8] The caption says 'pose recognition tasks' but the table reports segmentation models (MobileNet, PSANet, etc.); this appears to be a copy-paste error.
  3. [Eq. (3)] The notation is unclear: p_{iT/N} is used as a subscript and the sustainability function f is not explicitly defined. Please define f and the sampling indices precisely.
  4. [References] Reference [32] cites a genome-visualization paper for GcViT, but the model used in the experiments is a vision transformer; the citation is incorrect.
  5. [Experiments] No error bars, confidence intervals, or repeated runs are reported in any table; for an empirical metric comparison, this limits the strength of the conclusions.

Circularity Check

2 steps flagged · score 6.0 of 10

The claimed scale invariance of FMS and ASC is imposed by freely choosing alpha and wmax normalization, making the property true by construction rather than demonstrated; main-table ASC values also depend on an unreported discretization N.

  1. self definitional [Section 3.3 'Scale Invariance of Metrics on Electricity', Eq. (1)]
    "For the FMS method, assuming that we have determined the parameter α1 when the algorithm’s energy consumption is 1 kWh, then if the large model consumes 1000 kWh on large data, then according to Eqn. (1), We take α2 = α1/1000 to maintain identical energy consumption metrics. Therefore, FMS can make energy consumption with different levels scale invariant."

    The scale invariance asserted here is not a derived property of FMS; it is imposed by rescaling the free parameter α inversely with energy. Eq. (1), E(w)=e^{−αw}, and the instruction α2=α1/1000 make E(1000)=E(1) by construction. Since α is user-chosen and in the experiments is set per model/task (Sec. 4: 'α is set to 100 times the energy consumption (kWh) value at the 100th batch'), the cross-scale comparability is an input normalization, not an emergent characteristic. Claiming this as a property of the metric is therefore definitional.

  2. self definitional [Section 3.3, ASC normalization paragraph; conclusion]
    "It is worth noting that ASC needs to normalize the power value when calculating the area under the sustainability curve. Assuming that for algorithm A and algorithm B, wmax=1kWh and wmax=1000kWh, then the point wi=0.8 kWh on algorithm A will correspond to the point wj=800kWh on algorithm B, because these two points have the same weight."

    The ASC 'scale invariance' is obtained by dividing all energy values by wmax, but the metric value itself depends on wmax: Fig. 5 shows ASC increasing with wmax for all three models, and Table 7 shows ASC changing and rankings reversing with the discretization N. Thus the conclusion that 'Both metrics are scale-invariant with respect to energy consumption' is not a property of the definition; it is a normalization choice that the paper then treats as a demonstrated result.

full rationale

The core definitions of FMS and ASC are new and are evaluated on external benchmarks, so the paper is not wholly circular. However, the paper's key advertised property—scale invariance—is manufactured by free choices of α and wmax normalization, making the claim true by construction rather than by empirical demonstration. In addition, the main-table ASC values depend on an unreported discretization parameter N: Appendix B/Table 7 shows EfficientNet's ASC on ImageNet dropping from 0.5783 (N=100) to 0.2082 (N=500) and rankings flipping among models, yet the main tables report ASC values without specifying N, so the headline 'quantitative basis' is not reproducible from the paper's stated parameters. The SAM baseline [11] is authored by overlapping authors (Gowda, Jin), but it is used only as a comparison criterion, not as justification for the derivation, so it does not add circularity. The scale-invariance steps are the load-bearing circular elements, giving a partial circularity score of 6.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claims rest on several hand-set parameters (alpha, beta, wmax, N) and on assumptions about the smoothness of the performance-energy curve and the reliability of energy measurements. The largest issue is N: the paper's own ablations show the metric is not stable under changes in N.

free parameters (4)
  • alpha in E(w)=e^{-alpha w} = 100 times energy at 100th batch (classification, online/batch) or at 1000th iteration (segmentation, pose)
    Controls exponential decay of energy score; set by arbitrary heuristic per task.
  • beta in FMS = 1 (default), 0.5, 2 in ablation
    Balances weights between performance and energy; standard F-beta parameter, here chosen without prior justification.
  • wmax cutoff in ASC = 1 kWh in experiments; varied 0.8-1.8 in ablation
    Determines the energy horizon for area calculation; chosen ad hoc.
  • N intervals in ASC = Not stated in main experiments; 10, 50, 100, 500 in ablation
    Discretization for rectangular integration; strongly affects ASC values and rankings, yet unreported in main results.
assumptions (3)
  • ad hoc to paper Exponential decay E(w)=e^{-alpha w} is an appropriate normalization of training energy into [0,1].
    No justification given for exponential form; other mappings (e.g., linear, logistic) could be used.
  • domain assumption Energy consumption can be measured accurately and reproducibly during training.
    The paper does not specify the measurement tool (e.g., CodeCarbon, PowerAPI) or account for measurement noise.
  • domain assumption The performance-energy curve is smooth enough for rectangular integration to approximate the area.
    The paper uses discrete checkpoints; the observed sensitivity to N (Table 7) contradicts smoothness.
invented entities (2)
  • FMS (F-Measure on Sustainability)
    purpose: Composite score combining performance and exponentially normalized energy via harmonic mean.
    New metric not validated against an external ground truth of sustainability.
  • ASC (Area under Sustainability Curve)
    purpose: Area under the performance vs. normalized energy curve during training.
    New metric; its value and ranking ordering are sensitive to the discretization parameter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance is not All You Need: Sustainability Considerations for Algorithms." pith.science (2026). https://pith.science/paper/FNUWIGI2

@misc{pith2026250900045,
  author       = {Pith},
  title        = {Pith review of: Performance is not All You Need: Sustainability Considerations for Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNUWIGI2}},
  note         = {Machine review of arXiv:2509.00045}
}
read the original abstract

This work focuses on the high carbon emissions generated by deep learning model training, specifically addressing the core challenge of balancing algorithm performance and energy consumption. It proposes an innovative two-dimensional sustainability evaluation system. Different from the traditional single performance-oriented evaluation paradigm, this study pioneered two quantitative indicators that integrate energy efficiency ratio and accuracy: the sustainable harmonic mean (FMS) integrates accumulated energy consumption and performance parameters through the harmonic mean to reveal the algorithm performance under unit energy consumption; the area under the sustainability curve (ASC) constructs a performance-power consumption curve to characterize the energy efficiency characteristics of the algorithm throughout the cycle. To verify the universality of the indicator system, the study constructed benchmarks in various multimodal tasks, including image classification, segmentation, pose estimation, and batch and online learning. Experiments demonstrate that the system can provide a quantitative basis for evaluating cross-task algorithms and promote the transition of green AI research from theory to practice. Our sustainability evaluation framework code can be found here, providing methodological support for the industry to establish algorithm energy efficiency standards.

Figures

Figures reproduced from arXiv: 2509.00045 by the authors.

Figure 1
Figure 1. Area under sustainability curve: Energy consumption is normalized to [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Accuracy vs Energy on ImageNet for EfficientNet, SwinTransformer, and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. FMS vs Energy on ImageNet for EfficientNet, SwinTransformer, and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of accuracy, FMS, and ASC with power values on the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of ASC values of three models GcViT, SwinTransformer, and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Note that the horizontal axis represents the number of algorithm itera [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 6
Figure 6. Figure 6: The curve of FMS value changing with different [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 25 canonical work pages

  1. [11]

    arXiv preprint arXiv:2310.06522 (2023)

    Gowda, S.N., Hao, X., Li, G., Gowda, S.N., Jin, X., Sevilla-Lara, L.: Watt for what: Rethinking deep learning’s energy-performance relationship. arXiv preprint arXiv:2310.06522 (2023)

  2. [1]

    Alyamkin, S., Ardi, M., Berg, A.C., Brighton, A., Chen, B., Chen, Y., Cheng, H.P., Fan, Z., Feng, C., Fu, B., et al.: Low-power computer vision: Status, challenges, and opportunities. IEEE J. Emerg. Sel. Top. Circuits Sys9(2), 411–421 (2019)

  3. [2]

    arXiv preprint arXiv:2007.03051 (2021)

    Anthony, L., Kanding, B., Selvan, R.: Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051 (2021)

  4. [3]

    Springer-Verlag, Berlin, Heidelberg (2006)

    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer-Verlag, Berlin, Heidelberg (2006)

  5. [4]

    CoRR (2017)

    Chen, L., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. CoRR (2017)

  6. [5]

    https://github.com/open-mmlab/mmpose (2020)

    Contributors, M.: MMPose: Openmmlab pose estimation toolbox and benchmark. https://github.com/open-mmlab/mmpose (2020)

  7. [6]

    In: CVPR (2016)

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)

  8. [7]

    In: 2009 IEEE Conference on Computer Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009).https://doi.org/10.1109/CVPR.2009. 5206848

Show all 34 references
  1. [8]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy Alexey, e.a.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  2. [9]

    arXiv preprint arXiv:2309.14393 (2023)

    Faiz,A.,Kaneda,S.,Wang,R.,Osi,R.,Sharma,P.,Chen,F.,Jiang,L.:Llmcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393 (2023)

  3. [10]

    Sustainability 15(18) (2023)

    Fan, Z., Yan, Z., Wen, S.: Deep learning and artificial intelligence in sustainabil- ity: A review of sdgs, renewable energy, and environmental health. Sustainability 15(18) (2023)

  4. [12]

    arXiv preprint cs/0207029 (2002)

    Graves, A., Schmidhuber, J.: Bidirectional lstm networks for handwriting recogni- tion. arXiv preprint cs/0207029 (2002)

  5. [13]

    In: ICCV

    He, J., Deng, Z., Zhou, L., Wang, Y., Qiao, Y.: Adaptive pyramid context network for semantic segmentation. In: ICCV. pp. 7519–7528 (2019)

  6. [14]

    In: CVPR

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016) Sustainability Considerations for Algorithms 15

  7. [15]

    https://github.com/mlco2/codecarbon (2021), accessed: 2024-11- 23

    Henderson, P., Hu, J., Jong, W.S., Peyton, J., Prabhu, V., Rahtu, E., Xiao, R.: Codecarbon. https://github.com/mlco2/codecarbon (2021), accessed: 2024-11- 23

  8. [16]

    Information Sciences453, 50–65 (2018)

    Jin, X.B., Geng, G.G., Xie, G.S., Huang, K.: Approximately optimizing ndcg using pair-wise loss. Information Sciences453, 50–65 (2018)

  9. [17]

    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. . (2009)

  10. [18]

    In: ICML (2019)

    Lacoste, A., Diaby, A., Mielke, A., Parot, J., Chereau, G., Litz, P., Ricci, F.: Quantifying the carbon emissions of machine learning. In: ICML (2019)

  11. [19]

    In: Proceedings of the 1st International Workshop on Green and Sustainable Software

    Leclerc, G., Stévenin, V., Perache, M., Rouvoy, R., Merle, P., Seinturier, L.: Pow- erapi: A software library to monitor the energy consumed at the process level. In: Proceedings of the 1st International Workshop on Green and Sustainable Software. pp. 1–7 (2015)

  12. [20]

    In: ICCV

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV. pp. 10012–10022 (2021)

  13. [21]

    CSUR (2018)

    Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M.P., Shyu, M.L., Chen, S.C., Iyengar, S.S.: A survey on deep learning: Algorithms, techniques, and applications. CSUR (2018)

  14. [22]

    Kaggle (2015)

    Pozzolo, A.D.: Credit card fraud detection. Kaggle (2015)

  15. [23]

    Scientific Reports14 (2024)

    Rainio, O., Teuho, J., Klén, R.: Evaluation metrics and statistical tests for machine learning. Scientific Reports14 (2024)

  16. [24]

    Repository, U.M.L.: Image segments dataset.https://archive.ics.uci.edu/ml/ datasets/Image+Segmentation (1990)

  17. [25]

    Communications of the ACM (2020)

    Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green ai. Communications of the ACM (2020)

  18. [26]

    Foundations and Trends® in Machine Learning (2011)

    Shalev-Shwartz, S.: Online learning and online convex optimization. Foundations and Trends® in Machine Learning (2011)

  19. [27]

    In: UEMCON

    Sinha, D., El-Sharkawy, M.: Thin mobilenet: An enhanced mobilenet architecture. In: UEMCON. pp. 0280–0285. IEEE (2019)

  20. [28]

    In: AAAI

    Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: AAAI. vol. 33, pp. 10552–10560 (2019)

  21. [29]

    NeurIPS (2017)

    Vaswani, A.: Attention is all you need. NeurIPS (2017)

  22. [30]

    In: NeurIPS

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS. vol. 30 (2017)

  23. [31]

    TPAMI43(10), 3349–3364 (2020)

    Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. TPAMI43(10), 3349–3364 (2020)

  24. [32]

    BMC genomics (2020)

    Wilkey, A.P., Brown, A.V., Cannon, S.B., Cannon, E.K.: Gcvit: a method for interactive, genome-wide visualization of resequencing and snp array data. BMC genomics (2020)

  25. [33]

    arXiv preprint arXiv:2108.12617 (2021)

    Yu, H., Xu, Y., Zhang, J., Zhao, W., Guan, Z., Tao, D.: Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617 (2021)

  26. [34]

    In: ECCV (2018) 16 X

    Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C.C., Lin, D., Jia, J.: Psanet: Point-wise spatial attention network for scene parsing. In: ECCV (2018) 16 X. Li, C. Zhang et al. Appendix A Details on Tasks A.1 Image Classification The transform pipeline standardizes the raw images...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.