Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Framework architecture sets the energy cost of small-model coding agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 17:24 UTC pith:6IXO6ACI

load-bearing objection A genuinely new and careful energy measurement of SLM agent frameworks, but the causal framing ('architecture is the primary driver') outruns a between-codebase design with no ablation or LLM baseline. the 4 major comments →

arxiv 2512.09543 v2 pith:6IXO6ACI submitted 2025-12-10 cs.SE cs.AI

SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs

classification cs.SE cs.AI
keywords energy efficiencysmall language modelsagentic frameworksissue resolutionSWE-benchempirical studyresource consumptionSLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether small language models (SLMs) can power existing agentic issue-resolution frameworks efficiently on local hardware. Running four frameworks with two ≤4B-parameter models on 50 real GitHub issues, it finds near-zero success—only one configuration resolved 4% of tasks—while mean per-run energy varied 9.4-fold between frameworks using the same model. The authors argue that framework architecture, not the model, determines how much energy is burned, and that most energy goes into unproductive reasoning loops. The conclusion is that current frameworks, designed for large LLMs, need architectural changes—loop detection, guided exploration, and external verification—before SLMs become viable.

Core claim

With the same SLM and hardware, choosing one framework over another changed mean per-run energy from about 23 kJ (OpenHands) to about 216 kJ (AutoCodeRover), a 9.4x gap. Resolution was 0% for three of the four frameworks and 4% for AutoCodeRover paired with Qwen-3 1.7B, so nearly all consumed energy went into failed runs. Correlations show total energy tracks wall-clock duration (R=0.89) and output tokens (R=0.88), not memory usage. Failure signatures differ by architecture: chatty ReAct-style agents fall into repetition loops and context loss, while the deliberative AutoCodeRover burns energy in long runs that end in timeout. The paper's central claim is that these frameworks' passive orche

What carries the argument

The central object is the agentic framework's orchestration architecture: how the model is prompted to reason, call tools, and terminate. The study contrasts ReAct-style scaffolds (SWE-Agent, OpenHands, Mini SWE Agent) with AutoCodeRover's phased fault-localization-to-retrieval-to-patch pipeline. The comparison is carried by a controlled measurement methodology—fixed CPU/GPU hardware, RAPL and NVML energy traces with idle baseline subtracted, three repetitions per configuration across 1,200 total runs—and by correlation and failure-taxonomy analysis that links energy to runtime and token generation.

Load-bearing premise

The 9.4x energy gap is attributed to 'framework architecture' even though the four frameworks differ simultaneously in prompts, tool APIs, timeout behavior, and request orchestration, so the architecture attribution is not isolated from these confounded implementation details.

What would settle it

Run AutoCodeRover with the same prompt format, tool API, and timeout policy as OpenHands while keeping its phased pipeline; if the energy gap against OpenHands largely disappears, the gap is not architecture-driven. Alternatively, add a loop-detection mechanism to a single framework and check whether mean energy drops by the predicted amount while holding the model and issue set fixed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If architecture is the primary energy driver, swapping SLMs into existing LLM-oriented frameworks cannot be evaluated on accuracy alone; energy and failure modes must be measured per framework.
  • Low energy consumption is not a reliable efficiency signal: Mini SWE Agent's low-energy runs sometimes terminated prematurely or produced 'false positive' patches, so energy benchmarks need a correctness gate.
  • Energy benchmarking of agentic systems is feasible and reproducible on local hardware using RAPL/NVML with idle subtraction, enabling comparisons that cloud-based APIs obscure.
  • Future frameworks should include loop-breaking mechanisms, context filtering, and independent patch verification to avoid the observed wasted-energy loops.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the architectural attribution holds beyond these two models, energy-efficiency gains from redesigning orchestration—such as adding loop detection—may transfer to larger models too, making the finding relevant beyond the SLM niche.
  • A natural next experiment would ablate architecture from implementation: modify a single framework's loop-detection and context-filtering rules while holding prompts and tool APIs fixed, then re-measure the 9.4x gap; the paper does not perform this ablation.
  • The result suggests that 'green AI' arguments for SLMs are incomplete if frameworks are left unchanged; the environmental case for SLMs may hinge on framework redesign, not model choice alone.
  • Because resolution was near-zero, the paper's energy numbers mostly describe failed runs; the energy cost of a successfully resolved issue remains unmeasured, and a success-normalized metric (kJ per resolved issue) would be a stronger target for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a controlled experiment on energy consumption and effectiveness of four agentic issue-resolution frameworks (SWE-Agent, OpenHands, Mini SWE Agent, AutoCodeRover) paired with two small language models (Gemma-3 4B, Qwen-3 1.7B) on 50 tasks from SWE-bench Verified Mini. Each of the eight framework-model configurations was run three times per issue on fixed hardware (1,200 runs total), measuring CPU/GPU energy via RAPL/NVML with idle subtraction, plus runtime, tokens, LLM calls, memory, cost, and pass/fail via the official SWE-bench harness. The main findings are that only AutoCodeRover+Qwen resolved any tasks (mean 2/50, 4%); mean energy per run ranged from about 23 kJ (OpenHands) to 216 kJ (AutoCodeRover), a 9.4x gap; and energy correlated strongly with duration and output tokens (R ≈ 0.89 and 0.88). The authors conclude that framework architecture is the primary driver of energy consumption, that most energy is wasted in unproductive reasoning loops, and that the SLM limits success while the framework limits efficiency. They recommend SLM-aware architectures with loop-breaking, guided exploration, and external verification.

Significance. The paper has clear strengths: energy measurement on isolated fixed hardware with hardware counters and idle subtraction, use of the official SWE-bench evaluation harness, three repetitions per task, a stated replication package, and a first end-to-end hardware-level characterization of SLM-powered agentic issue-resolution frameworks. If the causal attribution were supported, the finding of near-zero success together with a 9.4x energy spread across frameworks would be practically important for sustainable, locally deployable agents. As it stands, the contribution is a careful measurement study and a reproducible methodology, but the headline causal claims about architecture go beyond what the comparison design can establish.

major comments (4)
  1. [§5.2, §6, §8] The central claim that 'framework architecture is the primary driver of energy consumption' and that 'architecture dictates how SLM reasoning failures manifest' is underdetermined by the design. The experiment varies four completely different codebases simultaneously: prompt templates, tool APIs, timeout handling, retry/stopping logic, request orchestration, and implementation language. No ablation, matched control, variance decomposition, or per-framework partial correlation is provided. The pooled correlation in Figure 3 (R ≈ 0.88) may reflect between-framework mean differences rather than an architectural mechanism. Please either soften the causal vocabulary to 'framework-level differences' and add this as an internal-validity threat in §7, or add experiments that isolate a specific mechanism.
  2. [§5.1, §6.3] The conclusion that 'the SLM's limited reasoning was the bottleneck for success' requires a larger-model baseline run under identical conditions. With all but one configuration resolving zero tasks, the near-zero rate could be attributable to the benchmark subset, framework misconfiguration, or harness-specific issues rather than to SLM capacity. Add at least one strong-LLM condition on the same tasks and frameworks, or substantially hedge this conclusion as applying only to the SLM configurations tested.
  3. [§7, §5.2] The paper acknowledges the near-zero resolution rate as a conclusion-validity threat, yet RQ2's causal mechanism statements rely on descriptive correlations and AI-assisted log analysis (Section 4.4) without hypothesis tests, confidence intervals, or inter-rater reliability metrics. With only three repetitions per issue and one to three total successes, the failure-mode attributions in Figure 5 and Lessons 1-3 should be presented as qualitative hypotheses, not as established effects. Report per-framework correlations and quantify uncertainty.
  4. [Table 1] Several rows report Total Energy inconsistent with the sum of CPU and GPU columns. For example, Mini SWE Agent/Qwen Mean: CPU 28.83 + GPU 28.96 = 57.79 kJ vs. Total 54.13 kJ; OpenHands/Qwen Mean: 20.18 + 5.59 = 25.77 kJ vs. Total 23.33 kJ. Since total energy is the headline quantity, reconcile the values and state explicitly whether 'Total' is measured independently or computed as a component sum.
minor comments (5)
  1. [Figure 1] Figure 1 says 'Run Experiments twice' but Section 4.3 states three full repetitions. Please correct the inconsistency.
  2. [§3.1] The sentence 'We evaluate four leading agentic frameworks: SWE-Agent, OpenHands, and AutoCodeRover' names only three frameworks; Mini SWE Agent should be listed.
  3. [Abstract] The abstract says 'over 150 runs per configuration,' but the design yields exactly 150 runs per configuration (50 tasks × 3 repetitions).
  4. [§5.2, Figure 3] Report p-values or confidence intervals for the reported correlations, or clearly label them as descriptive only. Also state whether the correlations pool all frameworks or are computed per framework.
  5. [§4.4, §7] The qualitative analysis used Gemini 2.5 Flash with manual verification by two authors. Please report the sample size of manually checked logs and any agreement measure, so readers can assess reliability.

Circularity Check

0 steps flagged

No significant circularity; energy and resolution outcomes are externally measured, not derived from the claims.

full rationale

This is an empirical measurement study rather than a derivation, and its central numbers do not reduce to their own inputs. Total energy is obtained from hardware counters (RAPL for CPU, NVML for GPU) with a measured idle baseline subtracted; pass/fail status is assigned by the official SWE-bench evaluation harness; duration, token counts, and memory are logged directly. The paper does not fit a parameter to part of the data and then present a closely related quantity as a prediction: the 9.4x energy comparison, the near-zero resolution rates, and the RQ2 correlations (R≈0.88 between energy and runtime/output tokens) are descriptive statistics over externally measured data. The qualitative log analysis in Section 4.4 is auxiliary explanation, not the source of the quantitative measurements. Although several references are co-authored by the present authors ([1], [5], [12], [18]), they are used as background framing and do not carry the load of the empirical conclusion; no uniqueness theorem or prior self-citation is invoked to force the interpretation. The main validity concern—that attributing the energy gap to 'framework architecture' is underdetermined because four different codebases vary simultaneously in prompts, tool APIs, timeouts, and orchestration—is a confound/internal-validity issue, not circularity: the measurements remain independent of the label attached to them. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters or invented entities are needed because the paper is an empirical measurement study. The central claims depend on domain assumptions about benchmark representativeness, instrumentation validity, the reliability of AI-assisted log analysis, and the causal attribution of energy differences to architecture. The last assumption is the most fragile and is also flagged as a red flag.

axioms (4)
  • domain assumption SWE-bench Verified Mini (50 tasks) preserves the difficulty distribution of the full SWE-bench Verified set.
    Section 3.1 cites the Hugging Face dataset card; no independent validation is provided, and the paper later acknowledges the small task count limits generalizability (Section 7).
  • domain assumption CPU/GPU energy measured by RAPL and NVML, with idle-power subtraction, isolates agent-induced energy consumption.
    Section 4.1; RAPL and NVML exclude RAM, storage, and motherboard power, and the subtraction assumes a constant idle draw under load.
  • domain assumption Failure-mode classifications derived from Gemini 2.5 Flash log processing, with two authors' manual spot-checks, are accurate.
    Section 4.4 and Section 7 acknowledge AI-assisted log analysis as a threat to validity; the manual verification is described but not quantified.
  • domain assumption Observed energy differences across frameworks can be attributed to architecture rather than to implementation-level confounds.
    Section 5.2 interprets cross-framework differences causally without an ablation or statistical model; the four codebases differ simultaneously in prompts, tools, defaults, and orchestration.

pith-pipeline@v1.3.0-alltime-deepseek · 11832 in / 16308 out tokens · 158428 ms · 2026-08-03T17:24:19.370681+00:00 · methodology

0 comments
read the original abstract

Context. LLM-based autonomous agents in software engineering rely on large, proprietary models, limiting local deployment. This has spurred interest in Small Language Models (SLMs), but their practical effectiveness and efficiency within complex agentic frameworks for automated issue resolution remain poorly understood. Goal. We investigate the performance, energy efficiency, and resource consumption of four leading agentic issue resolution frameworks when deliberately constrained to using SLMs. We aim to assess the viability of these systems for this task in resource-limited settings and characterize the resulting trade-offs. Method. We conduct a controlled evaluation of four leading agentic frameworks (SWE-Agent, OpenHands, Mini SWE Agent, AutoCodeRover) using two SLMs (Gemma-3 4B, Qwen-3 1.7B) on the SWE-bench Verified Mini benchmark. On fixed hardware, we measure energy, duration, token usage, and memory over 150 runs per configuration. Results. We find that framework architecture is the primary driver of energy consumption. The most energy-intensive framework, AutoCodeRover (Gemma), consumed 9.4x more energy on average than the least energy-intensive, OpenHands (Gemma). However, this energy is largely wasted. Task resolution rates were near-zero, demonstrating that current frameworks, when paired with SLMs, consume significant energy on unproductive reasoning loops. The SLM's limited reasoning was the bottleneck for success, but the framework's design was the bottleneck for efficiency. Conclusions. Current agentic frameworks, designed for powerful LLMs, fail to operate efficiently with SLMs. We find that framework architecture is the primary driver of energy consumption, but this energy is largely wasted due to the SLMs' limited reasoning. Viable low-energy solutions require shifting from passive orchestration to architectures that actively manage SLM weaknesses.

Figures

Figures reproduced from arXiv: 2512.09543 by Arihant Tripathy, Ch Pavan Harshit, Karthik Vaidhyanathan.

Figure 1
Figure 1. Figure 1: Overall study design showing the major phases. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Energy usage distribution across different scaffolds [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Token usage histograms for different frameworks. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Failure reasons stacked bar chart. issues reliably or efficiently. To explain these outcomes, we con￾ducted a qualitative analysis of the execution logs from the first run of each framework and model pair. This analysis, performed with the assistance of the Gemini 2.5 Flash model, revealed that the architectural flaws do not lie in the frameworks themselves, but in their implicit assumption of a highly cap… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 6 linked inside Pith

  1. [1]

    Radu Apsan, Vincenzo Stoico, Michel Albonico, Rudra Dhar, Karthik Vaid- hyanathan, and Ivano Malavolta. 2025. Generating Energy-Efficient Code via Large-Language Models – Where are we now?arXiv preprint arXiv:2509.10099 (2025)

  2. [2]

    Basili, Gianluigi Caldiera, and H

    Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. The Goal Question Metric Approach. InEncyclopedia of Software Engineering, John J. Marciniak (Ed.). Vol. 2. John Wiley & Sons, 528–532

  3. [3]

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Mu- ralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small Language Models are the Future of Agentic AI.arXiv preprint arXiv:2506.02153(2025)

  4. [4]

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, et al. 2025. Why Do Multi-Agent LLM Systems Fail?arXiv preprint arXiv:2503.13657(2025)

  5. [5]

    Luís Cruz, João Paulo Fernandes, Maja H. Kirkeby, Silverio Martínez-Fernández, June Sallou, Hina Anwar, Enrique Barba Roque, Justus Bogner, Joel Castaño, Fernando Castor, Aadil Chasmawala, Simão Cunha, Daniel Feitosa, Alexandra González, Andreas Jedlitschka, Patricia Lago, Henry Muccini, Ana Oprescu, Pooja Rani, João Saraiva, Federica Sarro, Raghavendra S...

  6. [6]

    {Vlad Andrei} Cursaru, Laura Duits, Joel Milligan, Damla Ural, {Berta Rodriguez} Sanchez, Vincenzo Stoico, and Ivano Malavolta. 2024. A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama. InQuality of Information and Communications Technology (Communications in Computer and Information Science), Antonia Bertolino, Jo...

  7. [7]

    Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388

  8. [8]

    Gemma Team, Google DeepMind. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv:2403.08295 https://arxiv.org/abs/2403.08295

  9. [9]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead.ACM Trans. Softw. Eng. Methodol.34, 5, Article 124 (May 2025), 30 pages. doi:10.1145/3712003

  10. [10]

    Marius Hobbhahn. 2025. SWE-bench Verified Mini. https://huggingface.co/ datasets/MariusHobbhahn/swe-bench-verified-mini. Hugging Face dataset; 50-instance subset of SWE-bench Verified

  11. [11]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

  12. [12]

    Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, and Karthik Vaidhyanathan. 2025. Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling. arXiv:2504.19277 [cs.AI] https: //arxiv.org/abs/2504.19277

  13. [13]

    Patricia Lago. 2019. Architecture design decision maps for software sustainability. In2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 61–64

  14. [14]

    Yue Liu, Sin Kit Lo, Qinghua Lu, Liming Zhu, Dehai Zhao, Xiwei Xu, Stefan Harrer, and Jon Whittle. 2025. Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents.Journal of Systems and Software220 (2025), 112278

  15. [15]

    Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Es- timating the Carbon Footprint of BLOOM, a 176B Parameter Language Model. Journal of Machine Learning Research24, 224 (2023). https://www.jmlr.org/ papers/volume24/23-0069/23-0069.pdf

  16. [16]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InForty- second International Conference on Machine Learning

  17. [17]

    Christoph Treude and Margaret-Anne Storey. 2025. Generative AI and Empirical Software Engineering: A Paradigm Shift. arXiv:2502.08108 [cs.SE] https://arxiv. org/abs/2502.08108

  18. [18]

    Karthik Vaidhyanathan and Henry Muccini. 2025. Software Architecture in the Age of Agentic AI. InSoftware Architecture. ECSA 2025 Tracks and Workshops: Limassol, Cyprus, September 15–19, 2025, Proceedings(Limassol, Cyprus). Springer- Verlag, Berlin, Heidelberg, 41–49. doi:10.1007/978-3-032-04403-7_5

  19. [19]

    Tina Vartziotis, Ippolyti Dellatolas, George Dasoulas, Maximilian Schmidt, Flo- rian Schneider, Tim Hoffmann, Sotirios Kotsopoulos, and Michael Keckeisen

  20. [20]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for A...

  21. [21]

    Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2025. Agents in software engineering: survey, landscape, and vision.Automated Software Engg. 32, 2 (Aug. 2025), 36 pages. doi:10.1007/s10515-025-00544-2

  22. [22]

    Grant Wilkins, Srinivasan Keshav, and Richard Mortier. 2025. Offline Energy- Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems.SIGENERGY Energy Inform. Rev.4, 5 (April 2025), 113–119. doi:10.1145/3727200.3727217

  23. [23]

    Jimenez, et al

    John Yang, Carlos E. Jimenez, et al . 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. InNeurIPS. https://papers. nips.cc/paper_files/paper/2024/file/5a7c947568c1b1328ccc5230172e1e7c-Paper- Conference.pdf

  24. [24]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR)

  25. [25]

    Yiheng Zhang et al. 2024. AutoCodeRover: Autonomous Program Improvement. InProceedings of the 32nd ACM ESEC/FSE. doi:10.1145/3650212.3680384

  26. [2024]

    In2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code)

    Learn to Code Sustainably: An Empirical Study on Green Code Generation. In2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). 30–37