{"id":"12c4236f-6296-4f15-a587-021518ae5800","arxiv_id":"2411.16340","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A framework for measuring user-side energy use of webmail services is proposed, with a five-run comparison of Gmail, Outlook, and Proton Mail that does not support the hypothesis that privacy-focused services are greener.","lead":"Researchers compared the energy use of Gmail, Outlook, and Proton Mail and found no clear evidence that privacy-friendly services are greener. They propose a measurement framework for the user side of online services, but their own tests are preliminary and mostly inconclusive.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The user-side ranking rests on unvalidated container-based energy attribution; a calibration test against a known load or power meter is needed before the Gmail/Proton comparison can be accepted.","rationale":"The reader correctly identifies the weakest assumption as the Green Metric Tool's container-based energy attribution. My stress-test confirms that this is the single most load-bearing concern: the paper's only concrete empirical result, the provider ranking, is an ordinal comparison of small energy differences, and the tool's attribution model is asserted without calibration or independent validation. The reader's CONDITIONAL verdict already reflects this requirement, so my analysis does not change the verdict; it sharpens the condition by specifying a ground-truth test that would settle whether the container attribution is trustworthy. I considered whether missing statistical tests should be the primary concern, but the measurement-validity issue is more fundamental: even with perfect statistical power, a biased attribution would invalidate the ranking, whereas with valid measurements the lack of reported statistics could be remedied by re-analysis. The paper's own limitation discussion does not address this attribution risk, instead asserting that user-side estimates are the most reliable, which is precisely the point that needs evidence. A concrete calibration test against a physical power meter would provide that evidence and is a reasonable condition for accepting the ranking claim.","tokens_in":6004,"tokens_out":3787,"duration_ms":35909,"concrete_test":"Run the GMT container-instrumented browser workload against a synthetic workload with known energy consumption (e.g., a fixed-duration CPU burn with a known instruction count) while measuring total system power with an external wall-plug power meter and recording container CPU time via cgroup accounting. Compute GMT's attributed energy and compare it to the externally measured energy for the container (wall-plug power × container CPU duty cycle); also repeat this at the observed Gmail/Proton power levels. If the error exceeds the between-provider differences reported in §3, the ranking is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in §3—that Gmail is slightly more energy-efficient and Proton consumes more energy, particularly in memory—depends entirely on the Green Metric Tool's container-scoped energy readings. The paper asserts this attribution is accurate ('By isolating the program in a container, the GMT creates a controlled environment that accurately evaluate[s] the specific energy consumption and data transfer attributable to the container') but provides no validation against a physical power meter, a calibrated workload, or an independent measurement tool. The reported differences are small ('slightly'), so systematic attribution errors—e.g., container overhead, CPU frequency scaling, or memory-energy estimation from indirect metrics—could easily overwhelm or invert the ranking. Section 3.1 claims 'on the user side, for which we have the most reliable estimates,' yet this reliability is assumed, not demonstrated. Without a ground-truth check, the Gmail/Proton ordering is not established as a property of the providers rather than of the measurement setup.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-level framework for assessing the environmental footprint of cloud-based office services, breaking the assessment into user-side, network/server-side, and embodied/end-of-life emissions. It applies the user-side part of the framework to Gmail, Outlook, and Proton Mail using Selenium-automated functional units and the Green Metric Tool for container-level energy monitoring. The experiments comprise five runs per functional unit and report no meaningful ad-blocker effect, while observing that Gmail is slightly more energy-efficient and Proton Mail consumes more energy, particularly in memory. The paper is explicitly cautious throughout, presenting these results as preliminary and listing limitations for each assessment layer.","tokens_in":6142,"tokens_out":4986,"duration_ms":45891,"significance":"The topic is timely, and the proposed framework is a useful structuring device for comparing the user-side energy behavior of online services. The authors are honest about the preliminary nature of the evidence and make their automation tool open source, which strengthens reproducibility. The use of external per-GB emission factors from Malmodin et al. rather than fitted parameters is a methodological plus. However, the paper's concrete empirical contribution, the ranking of the three webmail providers, is not yet established because it depends on unvalidated container-based energy attribution and on a very small number of runs without statistical reporting.","major_comments":[{"comment":"The user-side ranking of Gmail, Outlook, and Proton Mail rests entirely on the Green Metric Tool's container-level energy readings, yet the assertion that container isolation 'accurately evaluate[s] the specific energy consumption' is not validated against a physical power meter, a calibrated workload, or an independent measurement tool. Because the reported provider differences are only 'slightly' different, systematic attribution errors such as container overhead, CPU frequency scaling, or indirect memory-energy estimation could easily overwhelm or invert the ranking. The paper should either add a calibration experiment or explicitly demote the ranking claim to an illustration of the framework until such validation is provided.","section":"Section 3, Green Metric Tool paragraph"},{"comment":"The experiments consist of five runs per functional unit, but no variance, confidence intervals, or significance tests are reported. The sentence 'no significant differences were found' is therefore not a statistical conclusion; it is an informal observation. The paper should report per-run energy values with dispersion measures and apply an appropriate test, or at minimum show the distributions, for both the ad-blocker comparison and the provider comparison.","section":"Section 3, experimental results"},{"comment":"Standardizing functional units with wait statements so that all providers run for the same duration makes the ad-blocker comparison uninterpretable as a test of Pearce et al.'s time-savings result, as the paper itself acknowledges. This standardization may also affect the provider ranking, since equivalent user tasks such as reading three emails, replying, and logging out can take different wall-clock time and involve different amounts of work across providers. Re-running without artificial waits, or reporting both wall-clock time and energy per task, is needed before the provider differences can be attributed to service design rather than to the test harness.","section":"Section 3.1 and Section 2.1, Step 3"}],"minor_comments":[{"comment":"The text states that 'Six fundamental functional units were defined' but then enumerates seven items: Idle, Login, Logout, No attachment, Attachment, Reply, and Delete.","section":"Section 3, functional units"},{"comment":"The paper contains typographical artifacts in the title and header, including 'Environmenta l' and 'Office'; these should be corrected.","section":"Title and abstract"},{"comment":"References [7] and [8] carry the same title and URL; [8] should point to the Energy ID project page if that is intended.","section":"References [7] and [8]"},{"comment":"The claim that 'Gmail being slightly more energy-efficient and Proton consuming more energy, particularly in memory usage' should be accompanied by the actual energy values in Joules and their spread, so that 'slightly' is quantified.","section":"Section 3, provider comparison"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop extended abstract, and the authors are appropriately cautious in most places. My recommendation is driven by the fact that the concrete provider-ranking claim is load-bearing and currently rests on unvalidated measurement attribution and an unreported statistical basis. I would not reject; the framework and open-source artifact are worth publishing once the empirical claim is recalibrated or explicitly reframed as a preliminary case study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest, well-scoped workshop paper that combines existing tools into a user-side energy-assessment framework and applies it to Gmail, Outlook, and Proton Mail. The authors do not oversell their main hypothesis; they explicitly say the privacy-green link is unresolved. Their concrete positive claim is that Gmail is slightly more efficient and Proton heavier on memory, and that ad-blockers showed no significant effect.\n\nWhat I like: the framework decomposition is sensible, the choice of functional units is reasonable, and the limitations section actually engages with the hard problems—server-side opacity, network inelasticity, the ad-blocker's own cost, and the wait-statement distortion. They provide an artifact link, which is more than most. The citation pattern is appropriate: Pesari et al. is the right base, Pearce et al. is the right ad-blocker reference, and Malmodin et al. supplies the external emission factors without any parameter fitting to their own data.\n\nWhere it wobbles: the ranking depends entirely on Green Metric Tool's container-scoped energy readings, and there is no validation against a physical power meter or a known-load workload. The authors state the attribution is accurate on the authority of GMT's documentation. With only five runs per functional unit and no reported variance or statistical test, the \"slightly more efficient\" Gmail result could easily be measurement noise or container overhead rather than a property of the service. The ad-blocker null result is also weakened by the standardized wait statements; the paper acknowledges this. So the paper is a credible first step, not a confirmed finding.\n\nI agree with the stress-test note: the container-attribution question is load-bearing, and a calibration experiment is the missing piece.\n\nWho should read it: people in green software measurement and ICT carbon accounting, especially those thinking about how to attribute user-side energy to specific online services. It deserves referee time at a workshop; as a full journal paper it would need ground-truth calibration, more runs, and versioned artifacts. My call: send it to review, and ask the authors to validate the container attribution before any user-facing ranking is promoted.","headline":"An honest, well-scoped workshop paper extending user-side energy measurement to webmail; the concrete Gmail/Proton ranking is plausible but unvalidated against ground-truth power data.","tokens_in":6662,"tokens_out":2348,"would_cite":false,"duration_ms":21643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a Selenium-and-Green-Metric-Tool framework can rank webmail services by user-side energy consumption, and reports that in its measurements Gmail is slightly more efficient while Proton consumes more energy…","keywords":["privacy","security","environmental sustainability","energy consumption","webmail","ad-blocking","user-side measurement"],"falsifier":"Run the same five functional-unit scenarios for Gmail, Outlook, and Proton Mail while measuring total machine power with an external power meter; if the sum of the tool's per-container energy attributions does not track the meter within measurement error, the reported provider differences cannot be trusted.","tokens_in":5801,"feed_emoji":"🌱","tokens_out":6159,"duration_ms":48226,"temperature":0.7,"pith_summary":"The paper sets out to test whether privacy- and security-focused online services are greener than advertising-financed ones, using webmail as the case study. It builds a measurement framework that automates typical user tasks in a browser and uses container-isolated energy monitoring to attribute device power use to individual services. Applied to Gmail, Outlook, and Proton Mail, the framework finds no significant effect from ad-blocking and yields a preliminary user-side ranking in which Gmail is slightly more energy-efficient and Proton consumes more, especially in memory. The authors are explicit that this does not yet confirm or refute the main hypothesis that privacy-preserving services are greener, because server-side and embodied emissions are estimated from sector averages rather than measured.","feed_headline":"Gmail slightly beats Proton Mail in user-side energy test","feed_subtitle":"Automated browser scenarios rank three webmail providers on device energy; ad-blocking showed no measurable effect.","key_machinery":"The central object is the six-step user-side assessment pipeline: select comparable services, define functional units, automate them with Selenium, monitor with the Green Metric Tool's container isolation, convert energy to greenhouse-gas emissions via grid intensity, and compare emissions and traffic. The load-bearing mechanism is the Green Metric Tool's claim that enclosing a browser in a lightweight container allows the machine's embedded sensors to attribute energy and data transfer specifically to that container.","core_discovery":"On the paper's own terms, the discovery is that the user-side environmental footprint of an online office service can be decomposed into functional units (idle, login, logout, attachment, reply, delete), automated with Selenium, and monitored per-container with the Green Metric Tool, producing provider-level energy and data-traffic comparisons. In the limited runs, Gmail was slightly more energy-efficient and Proton consumed more energy, particularly in memory, while ad-blocker activation produced no statistically visible change. These results make it possible to say which tested webmail service is greener on the user side, but not yet to settle whether privacy-preserving business models are generally greener, because the dominant server-side and embodied emissions are estimated from sector averages rather than measured.","pith_inferences":["If container attribution is validated against physical power meters, the same framework could make energy scorecards for arbitrary web applications feasible.","The null ad-blocker result may be an artifact of webmail's low ad density; on ad-heavy sites the larger ad-blocker effects reported in earlier studies might resurface.","Because server-side and embodied emissions dominate the life cycle of cloud services, the privacy-greener hypothesis may ultimately be decided by data-center transparency rather than by user-side measurements.","A natural next experiment is to rerun the functional units without wait statements to see whether time savings from ad-blocking translate into energy savings, as the paper itself suggests."],"forward_implications":["If the framework is correct, anyone can rank webmail and other online office services by user-side energy with automated browser scenarios, without manual instrumentation.","Under these measurements, ad-blocking does not noticeably change user-side energy for webmail, suggesting the main user-side cost is the service's own code and data handling rather than displayed ads.","The functional-unit decomposition lets complex tasks be estimated by aggregating basic scenarios, which could extend to other office tools such as document editors and calendars.","The paper's user-side emissions estimates provide a concrete baseline against which future server-side transparency data could be compared."],"supporting_citations":[{"why":"Supplies the client-side energy and GHG assessment approach for advertising and tracking on news websites that the paper's framework extends.","marker":"[13]"},{"why":"Provides the Green Metric Tool used for container-isolated energy and network monitoring.","marker":"[7]"},{"why":"Quantifies energy savings from open-source ad-blockers and motivates the ad-blocker comparison.","marker":"[12]"},{"why":"Describes the mechanism by which online advertising monopolises network, data-centre, and device resources, framing the business-model hypothesis.","marker":"[16]"},{"why":"Supplies the sector-average network, server-side, and embodied emission factors per GB used in the non-user-side estimates.","marker":"[10]"},{"why":"Selenium is the automation tool that executes the functional-unit scenarios in the browser.","marker":"[2]"}],"fun_headline_variants":["User-side energy test: Gmail edges Proton Mail","Webmail energy: Gmail slightly greener than Proton","Ad-block no effect in user-side webmail energy test","Gmail beats Proton on user-side energy, ad-block no help"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire provider ranking rests on the Green Metric Tool's container isolation correctly assigning the computer's energy and network usage to the single service being tested, and the paper does not validate that attribution against a physical power meter or a known reference load.","fun_headline_variants_meta":{"raw":{"variants":["User-side energy test: Gmail edges Proton Mail","Webmail energy: Gmail slightly greener than Proton","Ad-block no effect in user-side webmail energy test","Gmail beats Proton on user-side energy, ad-block no help"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1273,"prompt_tokens":876,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":330}},"tokens_in":492,"tokens_out":397,"duration_ms":5891,"temperature":1.0,"reasoning_tokens":330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:13:06.242438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five functional-unit scenarios for Gmail, Outlook, and Proton Mail while measuring total machine power with an external power meter; if the sum of the tool's per-container energy attributions does not track the meter within measurement error, the reported provider differences cannot be trusted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the client-side energy and GHG assessment approach for advertising and tracking on news websites that the paper's framework extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Green Metric Tool used for container-isolated energy and network monitoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Quantifies energy savings from open-source ad-blockers and motivates the ad-blocker comparison."},{"cited_title":"Pärssinen, M","cited_arxiv_id":null,"evidence_quote":"Describes the mechanism by which online advertising monopolises network, data-centre, and device resources, framing the business-model hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Selenium is the automation tool that executes the functional-unit scenarios in the browser."}],"review_version":1}