REVIEW 4 major objections 5 minor 26 references
Framework architecture sets the energy cost of small-model coding agents.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 17:24 UTC pith:6IXO6ACI
load-bearing objection A genuinely new and careful energy measurement of SLM agent frameworks, but the causal framing ('architecture is the primary driver') outruns a between-codebase design with no ablation or LLM baseline. the 4 major comments →
SWEnergy: An Empirical Study on Energy Efficiency in Agentic Issue Resolution Frameworks with SLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
With the same SLM and hardware, choosing one framework over another changed mean per-run energy from about 23 kJ (OpenHands) to about 216 kJ (AutoCodeRover), a 9.4x gap. Resolution was 0% for three of the four frameworks and 4% for AutoCodeRover paired with Qwen-3 1.7B, so nearly all consumed energy went into failed runs. Correlations show total energy tracks wall-clock duration (R=0.89) and output tokens (R=0.88), not memory usage. Failure signatures differ by architecture: chatty ReAct-style agents fall into repetition loops and context loss, while the deliberative AutoCodeRover burns energy in long runs that end in timeout. The paper's central claim is that these frameworks' passive orche
What carries the argument
The central object is the agentic framework's orchestration architecture: how the model is prompted to reason, call tools, and terminate. The study contrasts ReAct-style scaffolds (SWE-Agent, OpenHands, Mini SWE Agent) with AutoCodeRover's phased fault-localization-to-retrieval-to-patch pipeline. The comparison is carried by a controlled measurement methodology—fixed CPU/GPU hardware, RAPL and NVML energy traces with idle baseline subtracted, three repetitions per configuration across 1,200 total runs—and by correlation and failure-taxonomy analysis that links energy to runtime and token generation.
Load-bearing premise
The 9.4x energy gap is attributed to 'framework architecture' even though the four frameworks differ simultaneously in prompts, tool APIs, timeout behavior, and request orchestration, so the architecture attribution is not isolated from these confounded implementation details.
What would settle it
Run AutoCodeRover with the same prompt format, tool API, and timeout policy as OpenHands while keeping its phased pipeline; if the energy gap against OpenHands largely disappears, the gap is not architecture-driven. Alternatively, add a loop-detection mechanism to a single framework and check whether mean energy drops by the predicted amount while holding the model and issue set fixed.
If this is right
- If architecture is the primary energy driver, swapping SLMs into existing LLM-oriented frameworks cannot be evaluated on accuracy alone; energy and failure modes must be measured per framework.
- Low energy consumption is not a reliable efficiency signal: Mini SWE Agent's low-energy runs sometimes terminated prematurely or produced 'false positive' patches, so energy benchmarks need a correctness gate.
- Energy benchmarking of agentic systems is feasible and reproducible on local hardware using RAPL/NVML with idle subtraction, enabling comparisons that cloud-based APIs obscure.
- Future frameworks should include loop-breaking mechanisms, context filtering, and independent patch verification to avoid the observed wasted-energy loops.
Where Pith is reading between the lines
- If the architectural attribution holds beyond these two models, energy-efficiency gains from redesigning orchestration—such as adding loop detection—may transfer to larger models too, making the finding relevant beyond the SLM niche.
- A natural next experiment would ablate architecture from implementation: modify a single framework's loop-detection and context-filtering rules while holding prompts and tool APIs fixed, then re-measure the 9.4x gap; the paper does not perform this ablation.
- The result suggests that 'green AI' arguments for SLMs are incomplete if frameworks are left unchanged; the environmental case for SLMs may hinge on framework redesign, not model choice alone.
- Because resolution was near-zero, the paper's energy numbers mostly describe failed runs; the energy cost of a successfully resolved issue remains unmeasured, and a success-normalized metric (kJ per resolved issue) would be a stronger target for future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a controlled experiment on energy consumption and effectiveness of four agentic issue-resolution frameworks (SWE-Agent, OpenHands, Mini SWE Agent, AutoCodeRover) paired with two small language models (Gemma-3 4B, Qwen-3 1.7B) on 50 tasks from SWE-bench Verified Mini. Each of the eight framework-model configurations was run three times per issue on fixed hardware (1,200 runs total), measuring CPU/GPU energy via RAPL/NVML with idle subtraction, plus runtime, tokens, LLM calls, memory, cost, and pass/fail via the official SWE-bench harness. The main findings are that only AutoCodeRover+Qwen resolved any tasks (mean 2/50, 4%); mean energy per run ranged from about 23 kJ (OpenHands) to 216 kJ (AutoCodeRover), a 9.4x gap; and energy correlated strongly with duration and output tokens (R ≈ 0.89 and 0.88). The authors conclude that framework architecture is the primary driver of energy consumption, that most energy is wasted in unproductive reasoning loops, and that the SLM limits success while the framework limits efficiency. They recommend SLM-aware architectures with loop-breaking, guided exploration, and external verification.
Significance. The paper has clear strengths: energy measurement on isolated fixed hardware with hardware counters and idle subtraction, use of the official SWE-bench evaluation harness, three repetitions per task, a stated replication package, and a first end-to-end hardware-level characterization of SLM-powered agentic issue-resolution frameworks. If the causal attribution were supported, the finding of near-zero success together with a 9.4x energy spread across frameworks would be practically important for sustainable, locally deployable agents. As it stands, the contribution is a careful measurement study and a reproducible methodology, but the headline causal claims about architecture go beyond what the comparison design can establish.
major comments (4)
- [§5.2, §6, §8] The central claim that 'framework architecture is the primary driver of energy consumption' and that 'architecture dictates how SLM reasoning failures manifest' is underdetermined by the design. The experiment varies four completely different codebases simultaneously: prompt templates, tool APIs, timeout handling, retry/stopping logic, request orchestration, and implementation language. No ablation, matched control, variance decomposition, or per-framework partial correlation is provided. The pooled correlation in Figure 3 (R ≈ 0.88) may reflect between-framework mean differences rather than an architectural mechanism. Please either soften the causal vocabulary to 'framework-level differences' and add this as an internal-validity threat in §7, or add experiments that isolate a specific mechanism.
- [§5.1, §6.3] The conclusion that 'the SLM's limited reasoning was the bottleneck for success' requires a larger-model baseline run under identical conditions. With all but one configuration resolving zero tasks, the near-zero rate could be attributable to the benchmark subset, framework misconfiguration, or harness-specific issues rather than to SLM capacity. Add at least one strong-LLM condition on the same tasks and frameworks, or substantially hedge this conclusion as applying only to the SLM configurations tested.
- [§7, §5.2] The paper acknowledges the near-zero resolution rate as a conclusion-validity threat, yet RQ2's causal mechanism statements rely on descriptive correlations and AI-assisted log analysis (Section 4.4) without hypothesis tests, confidence intervals, or inter-rater reliability metrics. With only three repetitions per issue and one to three total successes, the failure-mode attributions in Figure 5 and Lessons 1-3 should be presented as qualitative hypotheses, not as established effects. Report per-framework correlations and quantify uncertainty.
- [Table 1] Several rows report Total Energy inconsistent with the sum of CPU and GPU columns. For example, Mini SWE Agent/Qwen Mean: CPU 28.83 + GPU 28.96 = 57.79 kJ vs. Total 54.13 kJ; OpenHands/Qwen Mean: 20.18 + 5.59 = 25.77 kJ vs. Total 23.33 kJ. Since total energy is the headline quantity, reconcile the values and state explicitly whether 'Total' is measured independently or computed as a component sum.
minor comments (5)
- [Figure 1] Figure 1 says 'Run Experiments twice' but Section 4.3 states three full repetitions. Please correct the inconsistency.
- [§3.1] The sentence 'We evaluate four leading agentic frameworks: SWE-Agent, OpenHands, and AutoCodeRover' names only three frameworks; Mini SWE Agent should be listed.
- [Abstract] The abstract says 'over 150 runs per configuration,' but the design yields exactly 150 runs per configuration (50 tasks × 3 repetitions).
- [§5.2, Figure 3] Report p-values or confidence intervals for the reported correlations, or clearly label them as descriptive only. Also state whether the correlations pool all frameworks or are computed per framework.
- [§4.4, §7] The qualitative analysis used Gemini 2.5 Flash with manual verification by two authors. Please report the sample size of manually checked logs and any agreement measure, so readers can assess reliability.
Circularity Check
No significant circularity; energy and resolution outcomes are externally measured, not derived from the claims.
full rationale
This is an empirical measurement study rather than a derivation, and its central numbers do not reduce to their own inputs. Total energy is obtained from hardware counters (RAPL for CPU, NVML for GPU) with a measured idle baseline subtracted; pass/fail status is assigned by the official SWE-bench evaluation harness; duration, token counts, and memory are logged directly. The paper does not fit a parameter to part of the data and then present a closely related quantity as a prediction: the 9.4x energy comparison, the near-zero resolution rates, and the RQ2 correlations (R≈0.88 between energy and runtime/output tokens) are descriptive statistics over externally measured data. The qualitative log analysis in Section 4.4 is auxiliary explanation, not the source of the quantitative measurements. Although several references are co-authored by the present authors ([1], [5], [12], [18]), they are used as background framing and do not carry the load of the empirical conclusion; no uniqueness theorem or prior self-citation is invoked to force the interpretation. The main validity concern—that attributing the energy gap to 'framework architecture' is underdetermined because four different codebases vary simultaneously in prompts, tool APIs, timeouts, and orchestration—is a confound/internal-validity issue, not circularity: the measurements remain independent of the label attached to them. Therefore no circular step is present.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption SWE-bench Verified Mini (50 tasks) preserves the difficulty distribution of the full SWE-bench Verified set.
- domain assumption CPU/GPU energy measured by RAPL and NVML, with idle-power subtraction, isolates agent-induced energy consumption.
- domain assumption Failure-mode classifications derived from Gemini 2.5 Flash log processing, with two authors' manual spot-checks, are accurate.
- domain assumption Observed energy differences across frameworks can be attributed to architecture rather than to implementation-level confounds.
read the original abstract
Context. LLM-based autonomous agents in software engineering rely on large, proprietary models, limiting local deployment. This has spurred interest in Small Language Models (SLMs), but their practical effectiveness and efficiency within complex agentic frameworks for automated issue resolution remain poorly understood. Goal. We investigate the performance, energy efficiency, and resource consumption of four leading agentic issue resolution frameworks when deliberately constrained to using SLMs. We aim to assess the viability of these systems for this task in resource-limited settings and characterize the resulting trade-offs. Method. We conduct a controlled evaluation of four leading agentic frameworks (SWE-Agent, OpenHands, Mini SWE Agent, AutoCodeRover) using two SLMs (Gemma-3 4B, Qwen-3 1.7B) on the SWE-bench Verified Mini benchmark. On fixed hardware, we measure energy, duration, token usage, and memory over 150 runs per configuration. Results. We find that framework architecture is the primary driver of energy consumption. The most energy-intensive framework, AutoCodeRover (Gemma), consumed 9.4x more energy on average than the least energy-intensive, OpenHands (Gemma). However, this energy is largely wasted. Task resolution rates were near-zero, demonstrating that current frameworks, when paired with SLMs, consume significant energy on unproductive reasoning loops. The SLM's limited reasoning was the bottleneck for success, but the framework's design was the bottleneck for efficiency. Conclusions. Current agentic frameworks, designed for powerful LLMs, fail to operate efficiently with SLMs. We find that framework architecture is the primary driver of energy consumption, but this energy is largely wasted due to the SLMs' limited reasoning. Viable low-energy solutions require shifting from passive orchestration to architectures that actively manage SLM weaknesses.
Figures
Reference graph
Works this paper leans on
-
[1]
Radu Apsan, Vincenzo Stoico, Michel Albonico, Rudra Dhar, Karthik Vaid- hyanathan, and Ivano Malavolta. 2025. Generating Energy-Efficient Code via Large-Language Models – Where are we now?arXiv preprint arXiv:2509.10099 (2025)
Pith/arXiv arXiv 2025
-
[2]
Basili, Gianluigi Caldiera, and H
Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. The Goal Question Metric Approach. InEncyclopedia of Software Engineering, John J. Marciniak (Ed.). Vol. 2. John Wiley & Sons, 528–532
1994
-
[3]
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Mu- ralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small Language Models are the Future of Agentic AI.arXiv preprint arXiv:2506.02153(2025)
Pith/arXiv arXiv 2025
-
[4]
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, et al. 2025. Why Do Multi-Agent LLM Systems Fail?arXiv preprint arXiv:2503.13657(2025)
Pith/arXiv arXiv 2025
-
[5]
Luís Cruz, João Paulo Fernandes, Maja H. Kirkeby, Silverio Martínez-Fernández, June Sallou, Hina Anwar, Enrique Barba Roque, Justus Bogner, Joel Castaño, Fernando Castor, Aadil Chasmawala, Simão Cunha, Daniel Feitosa, Alexandra González, Andreas Jedlitschka, Patricia Lago, Henry Muccini, Ana Oprescu, Pooja Rani, João Saraiva, Federica Sarro, Raghavendra S...
arXiv 2025
-
[6]
{Vlad Andrei} Cursaru, Laura Duits, Joel Milligan, Damla Ural, {Berta Rodriguez} Sanchez, Vincenzo Stoico, and Ivano Malavolta. 2024. A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama. InQuality of Information and Communications Technology (Communications in Computer and Information Science), Antonia Bertolino, Jo...
2024
-
[7]
Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
Pith/arXiv arXiv 2025
-
[8]
Gemma Team, Google DeepMind. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv:2403.08295 https://arxiv.org/abs/2403.08295
Pith/arXiv arXiv 2024
-
[9]
Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead.ACM Trans. Softw. Eng. Methodol.34, 5, Article 124 (May 2025), 30 pages. doi:10.1145/3712003
doi:10.1145/3712003 2025
-
[10]
Marius Hobbhahn. 2025. SWE-bench Verified Mini. https://huggingface.co/ datasets/MariusHobbhahn/swe-bench-verified-mini. Hugging Face dataset; 50-instance subset of SWE-bench Verified
2025
-
[11]
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66
2024
-
[12]
Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, and Karthik Vaidhyanathan. 2025. Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling. arXiv:2504.19277 [cs.AI] https: //arxiv.org/abs/2504.19277
Pith/arXiv arXiv 2025
-
[13]
Patricia Lago. 2019. Architecture design decision maps for software sustainability. In2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 61–64
2019
-
[14]
Yue Liu, Sin Kit Lo, Qinghua Lu, Liming Zhu, Dehai Zhao, Xiwei Xu, Stefan Harrer, and Jon Whittle. 2025. Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents.Journal of Systems and Software220 (2025), 112278
2025
-
[15]
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Es- timating the Carbon Footprint of BLOOM, a 176B Parameter Language Model. Journal of Machine Learning Research24, 224 (2023). https://www.jmlr.org/ papers/volume24/23-0069/23-0069.pdf
2023
-
[16]
Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InForty- second International Conference on Machine Learning
2025
-
[17]
Christoph Treude and Margaret-Anne Storey. 2025. Generative AI and Empirical Software Engineering: A Paradigm Shift. arXiv:2502.08108 [cs.SE] https://arxiv. org/abs/2502.08108
arXiv 2025
-
[18]
Karthik Vaidhyanathan and Henry Muccini. 2025. Software Architecture in the Age of Agentic AI. InSoftware Architecture. ECSA 2025 Tracks and Workshops: Limassol, Cyprus, September 15–19, 2025, Proceedings(Limassol, Cyprus). Springer- Verlag, Berlin, Heidelberg, 41–49. doi:10.1007/978-3-032-04403-7_5
-
[19]
Tina Vartziotis, Ippolyti Dellatolas, George Dasoulas, Maximilian Schmidt, Flo- rian Schneider, Tim Hoffmann, Sotirios Kotsopoulos, and Michael Keckeisen
-
[20]
Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for A...
2025
-
[21]
Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2025. Agents in software engineering: survey, landscape, and vision.Automated Software Engg. 32, 2 (Aug. 2025), 36 pages. doi:10.1007/s10515-025-00544-2
-
[22]
Grant Wilkins, Srinivasan Keshav, and Richard Mortier. 2025. Offline Energy- Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems.SIGENERGY Energy Inform. Rev.4, 5 (April 2025), 113–119. doi:10.1145/3727200.3727217
arXiv 2025
-
[23]
Jimenez, et al
John Yang, Carlos E. Jimenez, et al . 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. InNeurIPS. https://papers. nips.cc/paper_files/paper/2024/file/5a7c947568c1b1328ccc5230172e1e7c-Paper- Conference.pdf
2024
-
[24]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR)
2023
-
[25]
Yiheng Zhang et al. 2024. AutoCodeRover: Autonomous Program Improvement. InProceedings of the 32nd ACM ESEC/FSE. doi:10.1145/3650212.3680384
arXiv 2024
-
[2024]
In2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code)
Learn to Code Sustainably: An Empirical Study on Green Code Generation. In2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code). 30–37
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.