{"id":"519f94d7-a0fc-4a09-aa5a-87086bf496eb","arxiv_id":"2412.17830","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"This is a review-style guide that organizes existing energy measurement methods, pitfalls, and reporting advice for computing and ML, without presenting new empirical results.","lead":"This paper is a beginner's guide to measuring and estimating how much electricity computers and machine learning programs use. It explains the main methods, the common mistakes, and how to turn energy numbers into carbon estimates.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proxy-estimate guidance in §2.5 rests on an unvalidated transfer of SPECPower/SERT curves to arbitrary ML workloads; if that transfer fails, the recommended fallback misleads beginners.","rationale":"The reader correctly identifies the transferability of benchmark-derived performance-to-power ratios as the weakest assumption. I agree with that assessment and with the UNVERDICTED classification of the paper as a whole, since it is a review-style guide rather than a research claim with falsifiable results. However, because the guide explicitly promises to orient beginners toward sound measurement and estimation, and because its only ready-to-use quantitative fallback is the proxy method in §2.5, this assumption is load-bearing. The paper does include caveats, but it still gives a concrete recommendation that may lead to systematically misleading estimates if the transfer does not hold. A single validation experiment would settle whether the concern lands: compare SPECPower-based estimates to wall-meter measurements for a representative ML workload. If the error is small, the concern dissolves; if it is large, the guide should add a validation requirement or downgrade the confidence of the proxy advice. Since this is a specific, addressable weakness rather than a reason to reject the entire guide, the verdict should be CONDITIONAL — accept only if the proxy recommendation is qualified or validated — rather than UNVERDICTED without conditions. No ad hominem is intended; the issue is the strength of the evidence base for one practical recommendation.","tokens_in":25626,"tokens_out":2555,"duration_ms":30300,"concrete_test":"On a single server, measure actual node power with a calibrated wall meter while running (a) the SPECPower workload and (b) a representative ML training or inference workload, each at controlled utilization levels (e.g., 25%, 50%, 75%, 100%). Estimate energy for the ML workload using the SPECPower curve for that server's architecture and TDP, as §2.5 recommends, then compare the estimate to the metered energy. If the relative error exceeds, say, 30% or is systematically biased for common ML workloads, the §2.5 fallback requires a stronger qualifier or a validation protocol before beginners are directed to rely on it.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that a beginner reading this guide will be adequately oriented to make sound measurement and interpretation choices. The most load-bearing assumption is in §2.5 and §2.5.1: when direct measurement is unavailable, the paper recommends estimating power by mapping resource utilization to a benchmark-derived performance-to-power ratio (SPECPower or SERT), and states that selecting a curve with the same CPU architecture and TDP is 'a solid starting point.' This transfer is not empirically justified in the paper. SPECPower and SERT use server-side Java HTTP workloads on specific bare-metal lab configurations; the power draw at a given utilization depends on instruction mix, memory/cache behavior, vector width, power-state residency, and attached peripherals, not only on architecture and TDP. The paper itself lists several caveats: the benchmark workload may not match the target workload, benchmark configurations are hard to replicate, and proxy measurements ignore temperature and clock state. Despite this, the concrete recommendation still endorses architecture/TDP matching as a sound fallback. For a beginner without access to a power meter, this is likely the only quantitative method they will use, and errors from it flow directly into reported energy and emissions estimates. The paper's own Scenario 3 used a SPECPower curve for CPU estimation and reported challenges aligning CPU and GPU estimates, but did not validate the curve against wall power. Thus the weakest link in the guide is not a lack of general caution but the absence of any evidence that the recommended transfer is accurate enough to support the guide's promise of sound estimates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a practical guide for measuring and estimating the energy consumption of computing and machine learning workloads. It introduces a measurement workflow, distinguishes system-, job-, and code-level goals, describes at-the-wall versus on-device measurement, discusses sampling, error sources, and power-management effects, and covers proxy estimation via benchmark-derived performance-to-power ratios and TDP. It also discusses converting energy to carbon emissions and offers reporting recommendations, three scenario walkthroughs, and an appendix of tools. The paper's central claim is that a beginner, after reading it, will be able to make sound measurement and interpretation choices when assessing the energy use of computing and ML workloads.","tokens_in":25908,"tokens_out":6029,"duration_ms":57707,"significance":"If the guidance is reliable, the paper fills an important practical gap: most ML researchers and developers do not currently incorporate energy measurement into their work, and existing guidance is scattered. The paper's strengths include a clear conceptual framework, explicit reporting checklists (Section 2.6.4), a balanced treatment of absolute versus marginal energy, and a useful catalogue of tools in Appendix 3. The authors are also appropriately modest about the limits of proxy estimates and about the lack of standardized protocols. Because the paper is pedagogical rather than a new measurement study, its value lies in whether its recommendations are clear and not misleading; the main risk is that some recommendations are stated more firmly than the evidence presented in the paper supports.","major_comments":[{"comment":"The recommendation that 'selecting a curve with the same architecture and TDP should be a solid starting point' is not validated for arbitrary ML workloads. The paper itself acknowledges in the bullet list in §2.5.1 that SPECPower and SERT use server-side Java workloads and that proxy measurements ignore temperature and clock state, but it does not present any evidence that architecture/TDP matching yields estimates within an acceptable error bound, nor does it quantify the error. This is load-bearing because this is the primary quantitative fallback for a beginner without a power meter. The guide should either provide a validation case (for example, comparing the SPECPower-based CPU estimate against wall-power measurement in Scenario 3, which is currently absent) or explicitly frame the recommendation as a rough order-of-magnitude estimate with guidance on how to report it as such. As written, the phrase 'solid starting point' may lead a beginner to treat proxy-derived numbers as more reliable than they are.","section":"§2.5.1"},{"comment":"The empirical calibration statement 'One publication did both physical measurement and Code Carbon measurement and estimated that the difference between the two on the hardware used is E (kWh) = 1.059 · codecarbon kWh' is given without a citation. Given the paper's own emphasis on transparent reporting (Section 2.6.4), this factor should cite the publication and specify the hardware, workload, and measurement conditions under which it was derived. As written, a reader could mistake it for a universal conversion factor, which would compound the very error sources the paper warns about.","section":"Appendix 3, item 5"}],"minor_comments":[{"comment":"The statement that memory modules 'do not require significant power to maintain stored data' is an oversimplification; DRAM refresh power is non-negligible and depends on access patterns. The sentence should be rephrased to avoid implying that memory power is always insignificant.","section":"§2.4.9"},{"comment":"TDP is described as 'the maximum amount of heat generated by the hardware component.' TDP is more accurately a design point for thermal dissipation, not a measured maximum power draw. The definition should be corrected to avoid reinforcing a common misconception.","section":"§2.5.2"},{"comment":"The suggestion to sample at intervals greater than 0.1 seconds is attributed to psutil without a citation; please add a reference to the psutil documentation or the relevant source.","section":"§2.4.5"},{"comment":"Greenwade (1993), which is about CTAN, appears to be a LaTeX template artifact and is unrelated to the paper's content; it should be removed.","section":"References"},{"comment":"There are several typos, including 'dissaggregated' (§2.2.1), 'refering' (§2.5), 'alterntive' (§2.4.9), 'gird' (§3), and 'standarized' (§4). A careful copyedit would improve readability.","section":"Throughout"},{"comment":"In the Reporting subsection, the sentence 'listing the tools and devices used for measurements, .' contains an extra comma; it should read 'listing the tools and devices used for measurements.'","section":"§2.6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a tutorial rather than a research contribution, so the bar for validation should be calibrated accordingly. The proxy-estimation concern in §2.5 is the most consequential issue: it is central to the paper's practical value, and the current wording ('solid starting point') is stronger than the evidence presented. If the authors add an explicit caveat with an uncertainty bound or a validation example, I would be comfortable moving toward acceptance. The missing citation for the Code Carbon conversion factor in Appendix 3 should also be fixed. The paper has many typos, but these are not blocking. Overall, the paper is a useful and honest primer that fills a real gap, provided the load-bearing proxy guidance is made appropriately cautious."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, genuinely useful beginner's guide to ML energy measurement, not a research contribution. It collects known pitfalls and organizes them well; the workflow framing and the scenario walkthroughs are the best parts. The proxy-estimation section (§2.5) is the weakest link, and the stress-test note is right: recommending architecture/TDP-matched SPECPower/SERT curves as a 'solid starting point' goes a bit beyond what the paper or the benchmarks can support. The guide itself lists the right caveats—benchmark workload mismatch, bare-metal lab conditions, ignored temperature/clock state—but then undercuts them by endorsing the transfer anyway. That is a moderate flaw, not a fatal one, because the guide frames proxies as fallbacks and tells readers to prefer direct measurement when possible. A beginner could still be misled into treating the estimate as accurate; the authors should either soften the claim or add a validation example.\n\nThe other soft spot is the unverified Code Carbon conversion factor in Appendix 3 (E = 1.059 × codecarbon kWh), which appears without error bars, citation, or hardware context. For a guide whose whole point is measurement accuracy, that one-liner is sloppy.\n\nWhat the paper does well: it distinguishes measurement goals (system/job/code), explains at-the-wall vs on-device, discusses sampling frequency, idle power, shared resources, and reporting transparency. The FAQ and scenario walkthroughs are concrete and practical. The self-citation to Tripp et al. and Bugbee et al. is appropriate—they are real example studies, and the guide's advice does not depend on them circularly.\n\nRecommendation: send to peer review. It's a tutorial, so reviewers should check for accuracy and completeness rather than novelty. The proxy-estimation wording and the Code Carbon conversion need fixing, but the core guidance is sound. Worth reading for anyone entering this subfield.","headline":"A genuinely useful beginner's guide to ML energy measurement that collects known pitfalls well, with a soft spot in its proxy-estimation recommendations and the unverified Code Carbon conversion factor.","tokens_in":26472,"tokens_out":1997,"would_cite":true,"duration_ms":21257,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that energy measurement for computing and machine learning is a series of scoped decisions, not a single physical reading, and provides a beginner's workflow for making those decisions soundly.","keywords":["energy measurement","power estimation","machine learning sustainability","performance-to-power ratio","thermal design power","at-the-wall measurement","sampling strategies","carbon footprint reporting"],"falsifier":"Run a representative ML training job on two servers with the same CPU architecture and TDP but different workload characteristics (for example, memory-bound versus compute-bound), estimate energy using a SPECPower- or SERT-style performance-to-power ratio, and compare against a calibrated wall meter. If the proxy estimate deviates from the wall-meter measurement by more than the paper's implied tolerance — or the deviation differs sharply between workloads — the matching-by-architecture rule is falsified for that class of workloads.","tokens_in":25481,"feed_emoji":"⚡","tokens_out":6073,"duration_ms":55315,"temperature":0.7,"pith_summary":"The paper argues that although energy is a physical property, turning it into a useful number for a computing or ML workload requires a chain of judgment calls: where to measure, how often, what to include, and how to fill gaps with estimates. It organizes these choices into three measurement levels — system, job/application, and code — and walks through common pitfalls at each level, such as idle power draw, cooling overhead, shared resources, sampling-rate effects, and component interactions. The paper's core practical claim is that a beginner can make sound energy estimates if they decide absolute-versus-marginal scope, match the measurement method to the question, and treat proxy estimates like performance-to-power ratios and thermal design power as fallbacks that carry real error. A sympathetic reader would come away with a checklist for planning, running, and reporting energy measurements, plus a call for better standards.","feed_headline":"How to measure the true energy cost of AI workloads","feed_subtitle":"A new guide walks through wall meters, on-device tools, sampling rates, and proxy estimates for computing and ML.","key_machinery":"The central mechanism is the measurement-scoping hierarchy, which separates questions into system, job/application, and code levels and maps them to a physical hierarchy from data center or facility down to component or device. The load-bearing analytical tools are the absolute-versus-marginal energy distinction, the in-band versus at-the-wall measurement distinction, and the performance-to-power ratio curve — a benchmark-derived mapping from CPU utilization to power draw — used as a proxy when direct measurement is unavailable.","core_discovery":"The paper's central claim is that sound energy measurement for ML and general computing is achievable for non-specialists, provided they understand that every measurement is a scoped estimate, not a raw physical readout. The paper establishes a workflow: define the question; identify relevant resources and measurement scope; choose between at-the-wall and on-device/in-band methods; select sampling rates that match the time scale of the workload; repeat measurements to characterize noise; and report hardware, software, methodology, and sources of error. It further claims that when direct measurement is impossible, proxy estimation using benchmarked performance-to-power ratios or published thermal design power (TDP) is a reasonable starting point, but with a higher error margin because the benchmark workload (often server-side Java) may not resemble the target workload. The paper supports these claims with three worked scenarios — system-level scheduler effects, job-level neural-network comparisons, and code-level optimization — and with a practical FAQ, tool list, and reporting guidance.","pith_inferences":["Beyond the paper: a natural testable extension is to validate proxy estimates against a few wall-meter spot checks on the actual deployment hardware; a small validation set would bound the proxy error and make the fallback recommendation trustworthy.","Beyond the paper: the paper's emphasis on active-idle power and processing stalls suggests that energy optimization of ML workloads may be dominated by data-movement and memory-hierarchy costs rather than raw compute, which would push optimization efforts toward batching and prefetching.","Beyond the paper: if proxy-based tools remain the default in ML sustainability reporting, the field could adopt a standardized reporting template that separates measured, estimated, and assumed components of an energy figure, reducing the risk of comparing apples to oranges."],"forward_implications":["A developer who follows the paper's checklist can produce energy numbers that are comparable across runs on the same system, and can state clearly what the numbers do and do not include.","Energy reported in model cards or papers should include runtime on hardware, measurement methodology, and sources of error; otherwise the number is not interpretable.","At-the-wall measurements capture the full system including conversion and cooling inefficiencies, while on-device measurements map cleanly to a workload but miss system overhead; the choice changes the conclusion.","Proxy estimates from benchmark ratios or TDP are acceptable fallbacks, but their error is workload-dependent, so they should not be treated as equivalent to direct measurement.","Because no universal standard exists, comparisons across different hardware and software environments require the same measurement methodology and scope."],"supporting_citations":[{"why":"Supplies the in-band versus out-of-band measurement distinction that frames the on-device versus at-the-wall discussion.","marker":"Grant et al. 2017"},{"why":"Introduces SPECpower, the benchmark whose performance-to-power ratios are the paper's main proxy-estimation example.","marker":"Lange 2009"},{"why":"Describes the SERT benchmark and the methodology for measuring power and energy efficiency used in proxy estimates.","marker":"Kistowski et al. 2018"},{"why":"Provides the concrete SPECpower result used as the example performance-to-power curve in Figure 3.","marker":"Standard Performance Evaluation Corporation 2024"},{"why":"Foundational training-energy study that motivates why ML researchers need to measure energy at all.","marker":"Strubell, Ganesh, and McCallum 2020"},{"why":"Example of a software-based measurement approach (Code Carbon) that the paper contrasts with physical meters and TDP-based estimates.","marker":"Luccioni, Jernite, and Strubell 2024"},{"why":"Example of a study using both Code Carbon and a physical energy meter, illustrating discrepancies between measurement approaches.","marker":"Wang et al. 2023"},{"why":"Supplies the instrumented HPC job-level neural-network energy dataset that Scenario 2 is built on.","marker":"Tripp et al. 2024"},{"why":"Supplies the system-level scheduler power-variability analysis used as Scenario 1.","marker":"Bugbee et al. 2017"}],"fun_headline_variants":["Measure AI energy without the guesswork","The practical guide to measuring ML power","Energy estimation for ML, demystified","How to get real energy numbers from AI","Cut through the fog of ML energy metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest link is the paper's own fallback advice in Section 2.5: when direct measurement is unavailable, it recommends matching a benchmark scenario by CPU architecture and TDP, yet the paper itself notes the benchmark workload (server-side Java) may not resemble the target ML workload.","fun_headline_variants_meta":{"raw":{"variants":["Measure AI energy without the guesswork","The practical guide to measuring ML power","Energy estimation for ML, demystified","How to get real energy numbers from AI","Cut through the fog of ML energy metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":2954,"prompt_tokens":873,"completion_tokens":2081,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":489,"tokens_out":2081,"duration_ms":16286,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:38:52.343191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a representative ML training job on two servers with the same CPU architecture and TDP but different workload characteristics (for example, memory-bound versus compute-bound), estimate energy using a SPECPower- or SERT-style performance-to-power ratio, and compare against a calibrated wall meter. If the proxy estimate deviates from the wall-meter measurement by more than the paper's implied tolerance — or the deviation differs sharply between workloads — the matching-by-architecture rule is falsified for that class of workloads.","supporting_citations":[],"review_version":1}