Pith. sign in

REVIEW 4 major objections 7 minor 39 references

Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems

T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read An LLM-based multi-agent pipeline refactors Haskell code, reporting lower cyclomatic complexity, runtime, and memory use while claiming to preserve behavior.

desk verdict Promising system architecture for Haskell refactoring, but the reported metrics are unsupported by missing reproducibility and correctness evidence. read the letter →

arxiv 2502.07928 v1 pith:PLXBWJFX submitted 2025-02-11 cs.SE

classification cs.SE
keywords GenerativeAILargeLanguageModelsCodeRefactorMulti-agentSystemHaskellProgrammingFunctionalCyclomaticComplexityPerformanceMetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an LLM-based multi-agent system can automate refactoring of Haskell codebases, a task usually left to manual work or rule-based tools. The authors built a pipeline of specialized agents—context analysis, verification, refactoring strategy, two refactoring passes, testing and validation, and debugging—and ran it on two Haskell codebases. They report that cyclomatic complexity fell by 13.64% and 47.06%, memory allocation fell by 4.17% and 41.73%, and runtime ticks fell from 4 to 2 and from 13 to 12 across the two codebases. The claim matters because if true, distributed LLM agents could lower the cost of maintaining functional programs without sacrificing correctness. The paper does not, however, report test results backing the functional-equivalence assertion.

What carries the argument

The load-bearing machinery is a sequential pipeline of specialized LLM-based agents, each paired with a verifier. A Code Context Agent parses the codebase and produces structural overviews; a Refactoring Analysis Agent computes cyclomatic complexity using $CC = E - N + 2P$, where $E$ is edges, $N$ is nodes, and $P$ is connected components; a Refactoring Strategy Agent proposes transformations; Refactor Agent 1 applies straightforward cleanups; Refactor Agent 2 performs advanced performance restructuring; a Testing and Validation Agent is supposed to confirm functional equivalence; and a Debug Agent resolves issues. The pipeline's design lets each stage refine or validate the previous stage's output, which is what the paper credits for both the metric gains and the claimed preservation of correctness.

What would settle it

Run the released system on the two codebases with the original test suite, or any suite that passes before refactoring, and check whether all tests still pass afterward; if any previously passing test fails, or a property-based equivalence check finds divergent outputs, the central claim of maintaining functional equivalence is false. Even without a suite, recomputing cyclomatic complexity on the post-refactor code and repeating the 0.01-second runtime measurements would show whether the reported 13.64% and 47.06% reductions and the 4-to-2 and 13-to-12 tick changes are stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that refactoring Haskell codebases can be automated by distributing the work across specialized LLM-based agents, and that this raises code quality without breaking behavior. On Codebase A, cyclomatic complexity dropped from 22 to 19 (13.64%); on Codebase B it dropped from 17 to 9 (47.06%). Runtime ticks fell from 4 to 2 (50%) and from 13 to 12 (7.69%), and memory allocation fell from 300,496 to 287,952 bytes (4.17%) and from 2,059,288 to 1,200,040 bytes (41.73%). HLint suggestions went from 2 to 3 on Codebase A and from 2 to 1 on Codebase B. The authors take these numbers as evidence that the agent pipeline optimizes Haskell's functional paradigms, and they state that the Testing and Validation Agent preserved functional equivalence, though the paper includes no test results.

Load-bearing premise

The entire safety claim rests on the Testing and Validation Agent actually checking that old and new code behave identically, but the paper gives no test results, pass/fail counts, or semantic comparisons to show that this check happened.

Editorial extensions

If this is right

  • If the reported numbers hold, an LLM-based multi-agent system can serve as a practical alternative to rule-based Haskell refactoring tools, automating tasks that currently require Haskell expertise.
  • The reductions in cyclomatic complexity and memory allocation imply that the refactoring pipeline can materially simplify control flow and reduce allocation hot spots in Haskell programs, not just make cosmetic changes.
  • The mixed HLint results indicate that stylistic improvement is not guaranteed by the pipeline; users should treat stylistic compliance as a separate objective from complexity and performance.
  • The system's modular agent design is claimed to scale to larger codebases through concurrency, which would make it suitable for industry-scale Haskell maintenance if the scalability claim is confirmed.
  • The public release of the system means other researchers can reproduce the two-codebase results and test the pipeline on additional Haskell projects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the functional-equivalence guarantee should be treated as unverified until test suites or semantic-comparison evidence are released; the only validation described is the existence of a Testing and Validation Agent, not its outputs.
  • Beyond the paper: the runtime measurements are coarse (0.01 seconds, tick counts of 4 to 2 and 13 to 12), so the reported 50% runtime improvement may partly reflect measurement granularity; a fair test would use longer-running benchmarks or more precise profiling.
  • Beyond the paper: the same architecture could be transplanted to other strongly typed functional languages such as OCaml or F#, since the agent roles are language-agnostic, but the paper does not test this.
  • A concrete extension would be to run the system on a held-out set of Haskell projects with a fixed pre/post test suite and property-based equivalence checks, turning the reported metric improvements into a falsifiable claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes an LLM-based multi-agent system for automating refactoring of Haskell codebases, consisting of specialized agents for context analysis, verification, refactoring strategy, code transformation, testing/validation, and debugging. The authors evaluate the system on two unnamed codebases and report improvements in cyclomatic complexity, runtime, and memory allocation, claiming that the refactored code 'maintains correctness and scalability.' The central claim is that the multi-agent design, combining LLMs with distributed task specialization, effectively improves Haskell code quality metrics.

Significance. If the claims were supported, the paper would offer a practical tool for automated Haskell refactoring, an area where existing tooling is largely rule-based. The multi-agent architecture with verifier agents is a plausible design, and the public GitHub release is a positive element. However, the current evaluation is not sufficient to establish the central claims: the correctness-preservation evidence is absent, the runtime measurements are statistically fragile, and the codebases are not identified. These gaps prevent the paper from making a credible empirical contribution as written.

major comments (4)
  1. [Sections 3.2, 3.3, 4] The 'maintaining correctness' claim in the abstract is unsupported. The Testing and Validation Agent is described in Section 3.2 as running unit, integration, and system tests, and Section 3.3 asserts that 'the Validation Agent ensured that refactored code maintained functional equivalence,' but Section 4 reports no test suite, no pass/fail counts, and no before/after output comparison. Without this evidence, the reported improvements in complexity, runtime, and memory could be the result of behavior-changing edits rather than safe refactoring, invalidating the central claim.
  2. [Table 1] The runtime results are measured as single tick counts with 1000 microseconds per tick: Codebase A goes from 4 to 2 ticks and Codebase B from 13 to 12 ticks. There are no repeated runs, no error bars, and no statistical test. At this granularity, a one-tick difference is within measurement noise, so the claimed 50% and 7.69% runtime improvements are not meaningful as presented.
  3. [Section 3.3] The evaluation is not reproducible because the two codebases are never identified. The text says 'For example, the refactoring of the Pandoc library's monadic parsing functions led to a 15% reduction in cyclomatic complexity,' but the Results section reports only Codebase A and Codebase B with reduction values of 13.64% and 47.06%, not 15%. This internal inconsistency, combined with the absence of repository names, sizes, or raw data, prevents any independent verification or replication.
  4. [Sections 3.2 and 3.4] The system description omits essential implementation details needed to reproduce the work: no LLM model or version, no prompt templates, no temperature or sampling parameters, no number of runs, and no version of GHC or HLint. A public repository link is given, but the manuscript itself does not state what was measured, on which code, or under what configuration, so the reported numbers cannot be traced to a concrete experimental setup.
minor comments (7)
  1. [Figure 1 and Section 1 text] The running text says the lambda \x -> x * x is replaced with the (^2) operator, but later says it is replaced with the (*2) operator; these are not equivalent (x*x versus x*2). The figure caption is also garbled, mixing explanatory text with code fragments.
  2. [Abstract] The abstract contains typographical errors: 'systems ability' should be 'system's ability' and 'Haskells functional paradigms' should be 'Haskell's functional paradigms'.
  3. [Section 2.1] The sentence beginning 'Further explored the challenges of refactoring functional programs' lacks a subject and is a fragment; it should be merged with the previous sentence or rewritten.
  4. [Section 2.2] The phrase 'This combination bridges theoretical concepts with practical marking a leap in multi-agent system capabilities' is ungrammatical and unclear.
  5. [Section 3.2] In the agent list, 'along with there responsibilities' should be 'along with their responsibilities'.
  6. [Table 1] The runtime rows list '0.01 secs' for both 4-tick and 13-tick measurements, but 4 ticks at 1000 us is 0.004 s and 13 ticks is 0.013 s; the displayed seconds values are inconsistently rounded, making the table misleading even at face value.
  7. [Section 7] The conclusion reports 'an average reduction of 20% in cyclomatic complexity' and 'a 15% improvement in memory allocation,' but the results in Table 2 are 13.64%/47.06% for complexity and 4.17%/41.73% for memory; these averages do not match, so the conclusion is internally inconsistent with the reported data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical before/after evaluation using standard external metrics, and its self-citations are background only.

full rationale

This paper is an empirical evaluation rather than a derivation, so the circularity burden is low. The reported reductions in cyclomatic complexity, runtime, and memory are measured before and after refactoring on the same codebases using standard external tools (McCabe's formula, GHC profiling with +RTS options, and HLint). The multi-agent system is explicitly designed to reduce those metrics, so measuring the same metrics as the objective is goal-consistency, not circular reasoning: the post-refactoring values are independent observations, not reconstructed from the system's inputs or prompts. The central correctness-preservation claim relies on the Testing and Validation Agent described in Section 3.2 as running 'unit, integration, and system tests,' and Section 3.3 asserts that 'the Validation Agent ensured that refactored code maintained functional equivalence.' No test suite, pass/fail counts, or before/after output comparisons are reported, which is a genuine evidence and reproducibility gap, but this is not a circularity: the claim is unsupported rather than equivalent to its inputs by construction. The paper's self-citations ([25]-[28] and other prior work by the same group) appear in background sections about multi-agent systems and do not carry the empirical Haskell refactoring claim; there is no imported uniqueness theorem, no fitted parameter renamed as a prediction, and no ansatz smuggled in via citation. Therefore no circular step can be quoted and exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about metric validity, codebase representativeness, and correctness preservation, none of which are independently verified in the paper. No invented entities are introduced.

assumptions (4)
  • domain assumption Cyclomatic complexity (E-N+2P) is a valid proxy for code maintainability.
    Used in Section 3.2 and 3.4 to quantify refactoring improvement; the paper does not justify this proxy, which is debatable for functional programs.
  • domain assumption GHC +RTS tick counts are sufficiently precise to measure runtime improvement.
    The paper computes 50% and 7.69% improvements from single measurements of 4 vs 2 ticks and 13 vs 12 ticks in Section 4.1, with no error analysis.
  • domain assumption The two selected codebases are representative of Haskell projects.
    Section 3.3 selects only two unnamed codebases through manual inspection and HLint filtering; external validity is acknowledged as a threat to validity.
  • domain assumption Refactored code preserves functional equivalence.
    Asserted in Section 3.3 without demonstrating test results or semantic comparison, despite the Testing and Validation Agent being described in Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems." pith.science (2026). https://pith.science/paper/PLXBWJFX

@misc{pith2026250207928,
  author       = {Pith},
  title        = {Pith review of: Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLXBWJFX}},
  note         = {Machine review of arXiv:2502.07928}
}
read the original abstract

We present a large language models (LLMs) based multi-agent system to automate the refactoring of Haskell codebases. The multi-agent system consists of specialized agents performing tasks such as context analysis, refactoring, validation, and testing. Refactoring improvements are using metrics such as cyclomatic complexity, run-time, and memory allocation. Experimental evaluations conducted on Haskell codebases demonstrate improvements in code quality. Cyclomatic complexity was reduced by 13.64% and 47.06% in the respective codebases. Memory allocation improved by 4.17% and 41.73%, while runtime efficiency increased by up to 50%. These metrics highlight the systems ability to optimize Haskells functional paradigms while maintaining correctness and scalability. Results show reductions in complexity and performance enhancements across codebases. The integration of LLMs based multi-agent system enables precise task execution and inter-agent collaboration, addressing the challenges of refactoring in functional programming. This approach aims to address the challenges of refactoring functional programming languages through distributed and modular systems.

Figures

Figures reproduced from arXiv: 2502.07928 by the authors.

Figure 1
Figure 1. Example: Haskell language before and after refac [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Research Methodology components, such as modules, functions, and their inter￾dependencies. Next, it generates visual and logical repre￾sentations, such as control flow diagrams, to illustrate the architecture of the codebase. These representations provide essential context for subsequent agents, ensuring that their analysis and refactoring tasks are precise and effective. • Context Verifier Agent: The Verifier Agent… view at source ↗
Figure 3
Figure 3. Cyclomatic Complexity Reduction for Codebase A [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Comparison of HLint Recommendations for Code [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Runtime and Memory Usage for Codebase A and B [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Comparison of Percentage Improvements Across [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

  1. [1]

    Chouarfia Abdallah and Hafida Bouziane. 2011. Dynamic Maintenance and Evolution of Critical Components-Based Software Using Multi Agent Systems. Computer and Information Science 4, 5 (2011), 78

  2. [2]

    S AyshwaryaLakshmi, SA Sahaaya Arul Mary, and S Shanmuga Vadivu. 2013. Agent based tool for topologically sorting badsmells and refactoring by analyzing complexities in source code. In2013 Fourth International Conference on Computing, Communications and Networking Technologies (ICCCNT) . IEEE, 1–7

  3. [3]

    Nils Baumgartner, Padma Iyenghar, Timo Schoemaker, and Elke Pulvermüller

  4. [4]

    Vitaly Bragilevsky. 2021. Haskell in Depth. Simon and Schuster

  5. [5]

    Christopher Brown, Huiqing Li, and Simon Thompson. 2011. An expression processor: a case study in refactoring Haskell programs. In Trends in Functional Programming: 11th International Symposium, TFP 2010, Norman, OK, USA, May 17-19, 2010. Revised Selected Papers 11 . Springer, 31–49

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

  7. [7]

    Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. 2024. Ex- ploring large language model based intelligent agents: Definitions, methods, and Distributed Approach to Haskell Based Applications Refactoring with LLMs Based Multi-Agent Systems Conference’17, July 2017, Washin...

  8. [8]

    Baldoino Fonseca dos Santos Neto, Márcio Ribeiro, Viviane Torres Da Silva, Christiano Braga, Carlos José Pereira De Lucena, and Evandro de Barros Costa

Show all 39 references
  1. [9]

    Ismael Figueroa, Paul Leger, and Hiroaki Fukuda. 2021. Which monads Haskell developers use: An exploratory study. Science of Computer Programming 201 (2021), 102523

  2. [10]

    Andy Gill and Graham Hutton. 2009. The worker/wrapper transformation. Journal of Functional Programming 19, 2 (2009), 227–251

  3. [11]

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680 (2024)

  4. [12]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

  5. [13]

    Zhenjiang Hu, John Hughes, and Meng Wang. 2015. How functional program- ming mattered. National Science Review 2, 3 (2015), 349–370

  6. [14]

    Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2023. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227 (2023)

  7. [15]

    Yu Huang. 2024. Levels of AI Agents: from Rules to Large Language Models. arXiv preprint arXiv:2405.06643 (2024)

  8. [16]

    Paul Hudak and Joseph H Fasel. 1992. A gentle introduction to Haskell. ACM Sigplan Notices 27, 5 (1992), 1–52

  9. [17]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515 (2024)

  10. [18]

    Huiqing Li, Simon Thompson, and Claus Reinke. 2005. The Haskell refactorer, HaRe, and its API. Electronic Notes in Theoretical Computer Science 141, 4 (2005), 29–34

  11. [19]

    Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320

  12. [20]

    Tom Mens and Tom Tourwé. 2004. A survey of software refactoring. IEEE Transactions on software engineering 30, 2 (2004), 126–139

  13. [21]

    Dominic Orchard and Tomas Petricek. 2014. Embedding effect systems in Haskell. In Proceedings of the 2014 ACM SIGPLAN Symposium on Haskell . 13–24

  14. [22]

    Simon L Peyton Jones and Philip Wadler. 1993. Imperative functional program- ming. In Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages. 71–84

  15. [23]

    Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bo- gomolov, Timofey Bryksin, and Danny Dig. 2024. Next-generation refactoring: Combining llm insights and ide capabilities for extract method. In 2024 IEEE International Conference on Software Maintenance a...

  16. [24]

    Lutz Prechelt. 2000. An empirical comparison of seven programming languages. Computer 33, 10 (2000), 23–29

  17. [25]

    Zeeshan Rasheed, Malik Abdul Sami, Kai-Kristian Kemell, Muhammad Waseem, Mika Saari, Kari Systä, and Pekka Abrahamsson. 2024. CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology. arXiv preprint arXiv:2402.01411 (2024)

  18. [26]

    Zeeshan Rasheed, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Wang Xiaofeng, Anh Nguyen Duc, and Pekka Abrahamsson. 2024. Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis. arXiv preprint arXiv:2402.01386 (2024)

  19. [27]

    Zeeshan Rasheed, Muhammad Waseem, Malik Abdul Sami, Kai-Kristian Kemell, Aakash Ahmad, Anh Nguyen Duc, Kari Systä, and Pekka Abrahamsson. 2024. Autonomous agents in software development: A vision paper. InInternational Conference on Agile Software Development . Springer Nature...

  20. [28]

    Zeeshan Rasheed, Muhammad Waseem, Kari Systä, and Pekka Abrahams- son. 2024. Large Language Model Evaluation Via Multi AI Agents: Prelimi- nary results. In ICLR 2024 Workshop on Large Language Model (LLM) Agents . https://openreview.net/forum?id=qMyFXpE888

  21. [29]

    Atsushi Shirafuji, Yusuke Oda, Jun Suzuki, Makoto Morishita, and Yutaka Watanobe. 2023. Refactoring programs using large language models with few- shot examples. In 2023 30th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 151–160

  22. [30]

    Shahbaz Siddeeq. 2025. Distributed-Approach-to-Haskell-Based- Applications-Refactoring-with-LLMs-Based-Multi-Agent-Systems. https://github.com/shahbazsiddeeq/Distributed-Approach-to-Haskell-Based- Applications-Refactoring-with-LLMs-Based-Multi-Agent-Systems Accessed: 31-Jan-2025

  23. [31]

    Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. InProceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering ....

  24. [32]

    Simon Thompson and Huiqing Li. 2013. Refactoring tools for functional languages. Journal of Functional Programming 23, 3 (2013), 293–350

  25. [33]

    Philip Wadler. 1992. The essence of functional programming. InProceedings of the 19th ACM SIGPLAN-SIGACT symposium on Principles of programming languages . 1–14

  26. [34]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt

  27. [35]

    Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review 10, 2 (1995), 115–152

  28. [36]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023)

  29. [37]

    In Generative AI for Effective Software Development

    Chatgpt prompt patterns for improving code quality, refactoring, require- ments elicitation, and software design. In Generative AI for Effective Software Development. Springer, 71–108

  30. [2015]

    Expert systems with applications 42, 3 (2015), 1652–1664

    AutoRefactoring: A platform to build refactoring agents. Expert systems with applications 42, 3 (2015), 1652–1664

  31. [2024]

    Electronics 13, 9 (2024), 1644

    AI-Driven Refactoring: A Pipeline for Identifying and Correcting Data Clumps in Git Repositories. Electronics 13, 9 (2024), 1644

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.