REVIEW 4 major objections 7 minor 39 references
Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems
T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read An LLM-based multi-agent pipeline refactors Haskell code, reporting lower cyclomatic complexity, runtime, and memory use while claiming to preserve behavior.
desk verdict Promising system architecture for Haskell refactoring, but the reported metrics are unsupported by missing reproducibility and correctness evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a sequential pipeline of specialized LLM-based agents, each paired with a verifier. A Code Context Agent parses the codebase and produces structural overviews; a Refactoring Analysis Agent computes cyclomatic complexity using $CC = E - N + 2P$, where $E$ is edges, $N$ is nodes, and $P$ is connected components; a Refactoring Strategy Agent proposes transformations; Refactor Agent 1 applies straightforward cleanups; Refactor Agent 2 performs advanced performance restructuring; a Testing and Validation Agent is supposed to confirm functional equivalence; and a Debug Agent resolves issues. The pipeline's design lets each stage refine or validate the previous stage's output, which is what the paper credits for both the metric gains and the claimed preservation of correctness.
What would settle it
Run the released system on the two codebases with the original test suite, or any suite that passes before refactoring, and check whether all tests still pass afterward; if any previously passing test fails, or a property-based equivalence check finds divergent outputs, the central claim of maintaining functional equivalence is false. Even without a suite, recomputing cyclomatic complexity on the post-refactor code and repeating the 0.01-second runtime measurements would show whether the reported 13.64% and 47.06% reductions and the 4-to-2 and 13-to-12 tick changes are stable.
Extended reading notes
Core claim
The paper's central claim is that refactoring Haskell codebases can be automated by distributing the work across specialized LLM-based agents, and that this raises code quality without breaking behavior. On Codebase A, cyclomatic complexity dropped from 22 to 19 (13.64%); on Codebase B it dropped from 17 to 9 (47.06%). Runtime ticks fell from 4 to 2 (50%) and from 13 to 12 (7.69%), and memory allocation fell from 300,496 to 287,952 bytes (4.17%) and from 2,059,288 to 1,200,040 bytes (41.73%). HLint suggestions went from 2 to 3 on Codebase A and from 2 to 1 on Codebase B. The authors take these numbers as evidence that the agent pipeline optimizes Haskell's functional paradigms, and they state that the Testing and Validation Agent preserved functional equivalence, though the paper includes no test results.
Load-bearing premise
The entire safety claim rests on the Testing and Validation Agent actually checking that old and new code behave identically, but the paper gives no test results, pass/fail counts, or semantic comparisons to show that this check happened.
Editorial extensions
If this is right
- If the reported numbers hold, an LLM-based multi-agent system can serve as a practical alternative to rule-based Haskell refactoring tools, automating tasks that currently require Haskell expertise.
- The reductions in cyclomatic complexity and memory allocation imply that the refactoring pipeline can materially simplify control flow and reduce allocation hot spots in Haskell programs, not just make cosmetic changes.
- The mixed HLint results indicate that stylistic improvement is not guaranteed by the pipeline; users should treat stylistic compliance as a separate objective from complexity and performance.
- The system's modular agent design is claimed to scale to larger codebases through concurrency, which would make it suitable for industry-scale Haskell maintenance if the scalability claim is confirmed.
- The public release of the system means other researchers can reproduce the two-codebase results and test the pipeline on additional Haskell projects.
Reading between the lines
- Beyond the paper: the functional-equivalence guarantee should be treated as unverified until test suites or semantic-comparison evidence are released; the only validation described is the existence of a Testing and Validation Agent, not its outputs.
- Beyond the paper: the runtime measurements are coarse (0.01 seconds, tick counts of 4 to 2 and 13 to 12), so the reported 50% runtime improvement may partly reflect measurement granularity; a fair test would use longer-running benchmarks or more precise profiling.
- Beyond the paper: the same architecture could be transplanted to other strongly typed functional languages such as OCaml or F#, since the agent roles are language-agnostic, but the paper does not test this.
- A concrete extension would be to run the system on a held-out set of Haskell projects with a fixed pre/post test suite and property-based equivalence checks, turning the reported metric improvements into a falsifiable claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an LLM-based multi-agent system for automating refactoring of Haskell codebases, consisting of specialized agents for context analysis, verification, refactoring strategy, code transformation, testing/validation, and debugging. The authors evaluate the system on two unnamed codebases and report improvements in cyclomatic complexity, runtime, and memory allocation, claiming that the refactored code 'maintains correctness and scalability.' The central claim is that the multi-agent design, combining LLMs with distributed task specialization, effectively improves Haskell code quality metrics.
Significance. If the claims were supported, the paper would offer a practical tool for automated Haskell refactoring, an area where existing tooling is largely rule-based. The multi-agent architecture with verifier agents is a plausible design, and the public GitHub release is a positive element. However, the current evaluation is not sufficient to establish the central claims: the correctness-preservation evidence is absent, the runtime measurements are statistically fragile, and the codebases are not identified. These gaps prevent the paper from making a credible empirical contribution as written.
major comments (4)
- [Sections 3.2, 3.3, 4] The 'maintaining correctness' claim in the abstract is unsupported. The Testing and Validation Agent is described in Section 3.2 as running unit, integration, and system tests, and Section 3.3 asserts that 'the Validation Agent ensured that refactored code maintained functional equivalence,' but Section 4 reports no test suite, no pass/fail counts, and no before/after output comparison. Without this evidence, the reported improvements in complexity, runtime, and memory could be the result of behavior-changing edits rather than safe refactoring, invalidating the central claim.
- [Table 1] The runtime results are measured as single tick counts with 1000 microseconds per tick: Codebase A goes from 4 to 2 ticks and Codebase B from 13 to 12 ticks. There are no repeated runs, no error bars, and no statistical test. At this granularity, a one-tick difference is within measurement noise, so the claimed 50% and 7.69% runtime improvements are not meaningful as presented.
- [Section 3.3] The evaluation is not reproducible because the two codebases are never identified. The text says 'For example, the refactoring of the Pandoc library's monadic parsing functions led to a 15% reduction in cyclomatic complexity,' but the Results section reports only Codebase A and Codebase B with reduction values of 13.64% and 47.06%, not 15%. This internal inconsistency, combined with the absence of repository names, sizes, or raw data, prevents any independent verification or replication.
- [Sections 3.2 and 3.4] The system description omits essential implementation details needed to reproduce the work: no LLM model or version, no prompt templates, no temperature or sampling parameters, no number of runs, and no version of GHC or HLint. A public repository link is given, but the manuscript itself does not state what was measured, on which code, or under what configuration, so the reported numbers cannot be traced to a concrete experimental setup.
minor comments (7)
- [Figure 1 and Section 1 text] The running text says the lambda \x -> x * x is replaced with the (^2) operator, but later says it is replaced with the (*2) operator; these are not equivalent (x*x versus x*2). The figure caption is also garbled, mixing explanatory text with code fragments.
- [Abstract] The abstract contains typographical errors: 'systems ability' should be 'system's ability' and 'Haskells functional paradigms' should be 'Haskell's functional paradigms'.
- [Section 2.1] The sentence beginning 'Further explored the challenges of refactoring functional programs' lacks a subject and is a fragment; it should be merged with the previous sentence or rewritten.
- [Section 2.2] The phrase 'This combination bridges theoretical concepts with practical marking a leap in multi-agent system capabilities' is ungrammatical and unclear.
- [Section 3.2] In the agent list, 'along with there responsibilities' should be 'along with their responsibilities'.
- [Table 1] The runtime rows list '0.01 secs' for both 4-tick and 13-tick measurements, but 4 ticks at 1000 us is 0.004 s and 13 ticks is 0.013 s; the displayed seconds values are inconsistently rounded, making the table misleading even at face value.
- [Section 7] The conclusion reports 'an average reduction of 20% in cyclomatic complexity' and 'a 15% improvement in memory allocation,' but the results in Table 2 are 13.64%/47.06% for complexity and 4.17%/41.73% for memory; these averages do not match, so the conclusion is internally inconsistent with the reported data.
Circularity Check
No significant circularity: the paper is an empirical before/after evaluation using standard external metrics, and its self-citations are background only.
full rationale
This paper is an empirical evaluation rather than a derivation, so the circularity burden is low. The reported reductions in cyclomatic complexity, runtime, and memory are measured before and after refactoring on the same codebases using standard external tools (McCabe's formula, GHC profiling with +RTS options, and HLint). The multi-agent system is explicitly designed to reduce those metrics, so measuring the same metrics as the objective is goal-consistency, not circular reasoning: the post-refactoring values are independent observations, not reconstructed from the system's inputs or prompts. The central correctness-preservation claim relies on the Testing and Validation Agent described in Section 3.2 as running 'unit, integration, and system tests,' and Section 3.3 asserts that 'the Validation Agent ensured that refactored code maintained functional equivalence.' No test suite, pass/fail counts, or before/after output comparisons are reported, which is a genuine evidence and reproducibility gap, but this is not a circularity: the claim is unsupported rather than equivalent to its inputs by construction. The paper's self-citations ([25]-[28] and other prior work by the same group) appear in background sections about multi-agent systems and do not carry the empirical Haskell refactoring claim; there is no imported uniqueness theorem, no fitted parameter renamed as a prediction, and no ansatz smuggled in via citation. Therefore no circular step can be quoted and exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Cyclomatic complexity (E-N+2P) is a valid proxy for code maintainability.
- domain assumption GHC +RTS tick counts are sufficiently precise to measure runtime improvement.
- domain assumption The two selected codebases are representative of Haskell projects.
- domain assumption Refactored code preserves functional equivalence.
Cite this review
Pith. "Pith review of Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems." pith.science (2026). https://pith.science/paper/PLXBWJFX
@misc{pith2026250207928,
author = {Pith},
title = {Pith review of: Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/PLXBWJFX}},
note = {Machine review of arXiv:2502.07928}
}
read the original abstract
We present a large language models (LLMs) based multi-agent system to automate the refactoring of Haskell codebases. The multi-agent system consists of specialized agents performing tasks such as context analysis, refactoring, validation, and testing. Refactoring improvements are using metrics such as cyclomatic complexity, run-time, and memory allocation. Experimental evaluations conducted on Haskell codebases demonstrate improvements in code quality. Cyclomatic complexity was reduced by 13.64% and 47.06% in the respective codebases. Memory allocation improved by 4.17% and 41.73%, while runtime efficiency increased by up to 50%. These metrics highlight the systems ability to optimize Haskells functional paradigms while maintaining correctness and scalability. Results show reductions in complexity and performance enhancements across codebases. The integration of LLMs based multi-agent system enables precise task execution and inter-agent collaboration, addressing the challenges of refactoring in functional programming. This approach aims to address the challenges of refactoring functional programming languages through distributed and modular systems.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Chouarfia Abdallah and Hafida Bouziane. 2011. Dynamic Maintenance and Evolution of Critical Components-Based Software Using Multi Agent Systems. Computer and Information Science 4, 5 (2011), 78
work page 2011
-
[2]
S AyshwaryaLakshmi, SA Sahaaya Arul Mary, and S Shanmuga Vadivu. 2013. Agent based tool for topologically sorting badsmells and refactoring by analyzing complexities in source code. In2013 Fourth International Conference on Computing, Communications and Networking Technologies (ICCCNT) . IEEE, 1–7
work page 2013
-
[3]
Nils Baumgartner, Padma Iyenghar, Timo Schoemaker, and Elke Pulvermüller
-
[4]
Vitaly Bragilevsky. 2021. Haskell in Depth. Simon and Schuster
work page 2021
-
[5]
Christopher Brown, Huiqing Li, and Simon Thompson. 2011. An expression processor: a case study in refactoring Haskell programs. In Trends in Functional Programming: 11th International Symposium, TFP 2010, Norman, OK, USA, May 17-19, 2010. Revised Selected Papers 11 . Springer, 31–49
work page 2011
-
[6]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)
arXiv 2021
-
[7]
Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. 2024. Ex- ploring large language model based intelligent agents: Definitions, methods, and Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems Conference’17, July 2017, Washin...
arXiv 2024
-
[8]
Baldoino Fonseca dos Santos Neto, Márcio Ribeiro, Viviane Torres Da Silva, Christiano Braga, Carlos José Pereira De Lucena, and Evandro de Barros Costa
Show all 39 references
-
[9]
Ismael Figueroa, Paul Leger, and Hiroaki Fukuda. 2021. Which monads Haskell developers use: An exploratory study. Science of Computer Programming 201 (2021), 102523
2021
-
[10]
Andy Gill and Graham Hutton. 2009. The worker/wrapper transformation. Journal of Functional Programming 19, 2 (2009), 227–251
2009
-
[11]
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680 (2024)
2024 arXiv
-
[12]
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79
2024
-
[13]
Zhenjiang Hu, John Hughes, and Meng Wang. 2015. How functional program- ming mattered. National Science Review 2, 3 (2015), 349–370
2015
-
[14]
Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2023. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227 (2023)
2023 arXiv
-
[15]
Yu Huang. 2024. Levels of AI Agents: from Rules to Large Language Models. arXiv preprint arXiv:2405.06643 (2024)
2024 arXiv
-
[16]
Paul Hudak and Joseph H Fasel. 1992. A gentle introduction to Haskell. ACM Sigplan Notices 27, 5 (1992), 1–52
1992
-
[17]
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515 (2024)
2024 arXiv
-
[18]
Huiqing Li, Simon Thompson, and Claus Reinke. 2005. The Haskell refactorer, HaRe, and its API. Electronic Notes in Theoretical Computer Science 141, 4 (2005), 29–34
2005
-
[19]
Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320
1976
-
[20]
Tom Mens and Tom Tourwé. 2004. A survey of software refactoring. IEEE Transactions on software engineering 30, 2 (2004), 126–139
2004
-
[21]
Dominic Orchard and Tomas Petricek. 2014. Embedding effect systems in Haskell. In Proceedings of the 2014 ACM SIGPLAN Symposium on Haskell . 13–24
2014
-
[22]
Simon L Peyton Jones and Philip Wadler. 1993. Imperative functional program- ming. In Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages. 71–84
1993
-
[23]
Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bo- gomolov, Timofey Bryksin, and Danny Dig. 2024. Next-generation refactoring: Combining llm insights and ide capabilities for extract method. In 2024 IEEE International Conference on Software Maintenance a...
2024
-
[24]
Lutz Prechelt. 2000. An empirical comparison of seven programming languages. Computer 33, 10 (2000), 23–29
2000
-
[25]
Zeeshan Rasheed, Malik Abdul Sami, Kai-Kristian Kemell, Muhammad Waseem, Mika Saari, Kari Systä, and Pekka Abrahamsson. 2024. CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology. arXiv preprint arXiv:2402.01411 (2024)
2024 arXiv
-
[26]
Zeeshan Rasheed, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Wang Xiaofeng, Anh Nguyen Duc, and Pekka Abrahamsson. 2024. Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis. arXiv preprint arXiv:2402.01386 (2024)
2024
-
[27]
Zeeshan Rasheed, Muhammad Waseem, Malik Abdul Sami, Kai-Kristian Kemell, Aakash Ahmad, Anh Nguyen Duc, Kari Systä, and Pekka Abrahamsson. 2024. Autonomous agents in software development: A vision paper. InInternational Conference on Agile Software Development . Springer Nature...
2024
-
[28]
Zeeshan Rasheed, Muhammad Waseem, Kari Systä, and Pekka Abrahams- son. 2024. Large Language Model Evaluation Via Multi AI Agents: Prelimi- nary results. In ICLR 2024 Workshop on Large Language Model (LLM) Agents . https://openreview.net/forum?id=qMyFXpE888
2024
-
[29]
Atsushi Shirafuji, Yusuke Oda, Jun Suzuki, Makoto Morishita, and Yutaka Watanobe. 2023. Refactoring programs using large language models with few- shot examples. In 2023 30th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 151–160
2023
-
[30]
Shahbaz Siddeeq. 2025. Distributed-Approach-to-Haskell-Based- Applications-Refactoring-with-LLMs-Based-Multi-Agent-Systems. https://github.com/shahbazsiddeeq/Distributed-Approach-to-Haskell-Based- Applications-Refactoring-with-LLMs-Based-Multi-Agent-Systems Accessed: 31-Jan-2025
2025
-
[31]
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. InProceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering ....
2020
-
[32]
Simon Thompson and Huiqing Li. 2013. Refactoring tools for functional languages. Journal of Functional Programming 23, 3 (2013), 293–350
2013
-
[33]
Philip Wadler. 1992. The essence of functional programming. InProceedings of the 19th ACM SIGPLAN-SIGACT symposium on Principles of programming languages . 1–14
1992
-
[34]
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt
-
[35]
Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review 10, 2 (1995), 115–152
1995
-
[36]
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023)
2023 arXiv
-
[37]
In Generative AI for Effective Software Development
Chatgpt prompt patterns for improving code quality, refactoring, require- ments elicitation, and software design. In Generative AI for Effective Software Development. Springer, 71–108
-
[2015]
Expert systems with applications 42, 3 (2015), 1652–1664
AutoRefactoring: A platform to build refactoring agents. Expert systems with applications 42, 3 (2015), 1652–1664
2015
-
[2024]
Electronics 13, 9 (2024), 1644
AI-Driven Refactoring: A Pipeline for Identifying and Correcting Data Clumps in Git Repositories. Electronics 13, 9 (2024), 1644
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.