Pith. sign in

REVIEW 1 major objections 1 minor 27 references

Can LLMs Replace Humans During Code Chunking?

T0 review · 1 major / 1 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims LLM-made code partitions yield documentation up to 20% more factual and 10% more useful than human-made partitions.

desk verdict The RQ1 result on LLM partitioning is credible, but the 'up to 20% more factual' claim rests on unvalidated LLM self-evaluation and should not be cited as-is. read the letter →

arxiv 2506.19897 v1 pith:PXNXBW6S submitted 2025-06-24 cs.SE cs.AI

Christopher Glasz , Emily Escamilla , Eric O. Scott , Anand Patel , Jacob Zimmer , Colin Diggs , Michael Doyle , Scott Rosen
show 9 more authors
Nitin Naik Justin F. Brunelle Samruddhi Thaker Parthav Poudel Arun Sridharan Amit Madan Doug Wendt William Macke Thomas Schill
This is my paper
classification cs.SEcs.AI
keywords largelanguagemodelslegacysystemmodernizationcodepartitioningchunkingcommentgenerationMUMPSassemblydocumentationquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models need code to fit inside their context windows, so modernizing huge legacy codebases requires splitting files into chunks. This paper asks who should do the splitting: human experts or the LLM itself. Across two legacy languages (assembly-like ALC and MUMPS), four LLMs, and six chunking strategies, the authors find that LLM-chosen partition points align closely with human expert partitions and that comments generated from LLM partitions are rated up to 20% more factual and up to 10% more useful than comments generated from human partitions. The conclusion is that LLM partitioning can replace human partitioning in LLM-aided modernization, which matters because legacy government systems are enormous and subject-matter experts are scarce.

What carries the argument

The load-bearing procedure is LLM Partitioning: the model is prompted to split a file into relatively self-contained logical blocks, ideally complete subroutines, and to return the breakpoints as JSON, with an optional token limit enforced by re-querying the model with feedback when a chunk is too long. Its effectiveness is measured through a pipeline that strips existing comments, inserts unique module markers, generates module-level comments with four LLMs, and grades those comments with GPT-4o acting as a hard grader on 0-100 scales for completeness, hallucination, readability, and usefulness, averaging ten independent evaluations per comment.

What would settle it

Have a panel of human experts in MUMPS and ALC blindly rate a sample of generated comments from LLM Partitions and Human Partitions using the same rubric; if the human-rated factualness advantage is below the reported 20% or absent, the central conclusion would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that the way code is chunked materially affects the quality of downstream documentation generation, and that LLM partitioning is the most consistently strong chunking method. For research question one, LLM-chosen partitions most closely matched subject-matter-expert partitions on both the MUMPS and ALC corpora, with the exception of Single Module chunking on MUMPS. For research question two, LLM partitioning produced comments that scored at least as well as, and usually better than, human partitioning across all four LLMs and all four evaluation metrics: completeness, hallucination/factualness, readability, and usefulness. The headline numbers are up to 20% higher factualness (i.e., fewer hallucinated statements) and up to 10% higher usefulness for LLM-created partitions over human-created partitions, leading the authors to conclude that LLMs can serve as suitable replacements for human partitioning during LLM-aided modernization of large legacy codebases.

Load-bearing premise

The comparison of comment quality assumes that GPT-4o's 0-100 ratings for completeness, hallucination, readability, and usefulness are trustworthy proxies for what human experts would say; the paper does not validate these scores against human judgments.

Editorial extensions

If this is right

  • LLM partitioning can stand in for human expert partitioning when preparing large legacy code files for LLM-based modernization, reducing the need for scarce subject-matter experts.
  • Chunking choices should be tuned per model rather than assumed universal; the best method for GPT-4o differed from the best for Claude 3 Sonnet, Mixtral, and Llama 3.
  • More context does not always help: larger chunks improved factualness on MUMPS but not usefulness on ALC, so token limits should be chosen with the target language in mind.
  • LLM partitioning remains viable where abstract syntax tree parsers are unavailable or too costly, since it does not require language-specific syntax tooling.
  • Documentation generated from LLM partitions is consistently high quality across models and metrics, making comment generation more reliable for downstream code conversion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A human-rating study is the natural next test: if expert raters do not reproduce the 20% factualness advantage, the reported gain could partly reflect evaluator bias rather than a true quality difference.
  • The same LLM-partitioning recipe could carry over to other poorly structured large codebases, not just legacy languages, whenever function boundaries are hard to detect automatically.
  • Because the authors observed similar behavior on government systems but could only publish open-source corpora, the public evidence for the strongest form of the conclusion rests on an assumed similarity between those corpora and classified workloads.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper investigates how to partition large legacy code files (MUMPS and ALC) so that LLMs can generate useful module-level documentation when the full file exceeds context-window limits. It compares naive, structure-based, expert human, and LLM-generated partitions, evaluating (RQ1) how closely each method reproduces human SME partition points and (RQ2) the quality of comments generated by GPT-4o, Claude 3 Sonnet, Mixtral, and Llama 3 under each partitioning scheme. The main quantitative claims are that LLM partitions align closely with human partitions and that LLM-created partitions yield comments that are 'up to 20% more factual and up to 10% more useful' than human-created partitions, leading to the conclusion that LLMs can replace human partitioning during LLM-aided modernization.

Significance. If the central claims held, the paper would make a practically useful contribution: legacy code modernization often lacks AST parsers and subject-matter experts, so an automated, LLM-based partitioning method that improves downstream documentation quality would be valuable. The RQ1 analysis is a genuine empirical contribution and the experimental setup is substantial (two corpora, four LLMs, sixteen chunking configurations, and a large evaluation campaign). However, the headline RQ2 conclusion is currently unsupported because the documentation-quality scores come from LLM self-evaluation without human validation, and the 'up to 20%' figures are unadjusted maxima over many comparisons. The significance of the paper is therefore conditional on addressing these measurement-validity concerns.

major comments (1)
  1. [Section III-D2] The evaluation section is ambiguous about which model produced the scores used in Figures 2-5. It says 'we prompted GPT-4o and Claude 3 Sonnet to evaluate' but the reported request count (1.28 million to GPT-4o) and the ICC values are only for GPT-4o. If Claude 3 Sonnet also served as an evaluator, its agreement and any validation should be reported separately; if not, the text should be corrected. This matters because the central comparison may depend on the choice of a single judge.
minor comments (1)
  1. [Figures 2-5] The box plots would be easier to interpret with error bars or annotated significance markers; as they stand, the reader cannot assess the variability behind the 'up to' claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain; LLM-as-judge is a validity limitation, not a circular reduction.

full rationale

The paper's argument is empirical: partitioning methods are manipulated variables and comment-quality scores are measured outcomes (Section III-D2). RQ1 uses human SME partitions as ground truth, so the claim that LLM partitions resemble human partitions is an external comparison, not a construction. RQ2's conclusion that LLM-created partitions yield comments 'up to 20% more factual and up to 10% more useful' is based on scores produced by GPT-4o (and Claude 3 Sonnet), which the paper itself labels 'LLM self-evaluation scores' (Section IV-B). This is a measurement-validity threat: the judge is the same class of model as the generators, the rubric is not validated against human experts, and the headline percentages are maxima over many comparisons without correction. But no load-bearing step reduces by definition to its inputs: the outcome metric is not defined in terms of the partitioning input, no parameter is fitted and then renamed a prediction, and the only self-citation (Ref. [8], used to motivate the importance of comments) is background rather than load-bearing. The central comparison is an empirical measurement with an unvalidated instrument, which is a correctness risk rather than a circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or new entities. The core assumptions are the validity of LLM-based comment evaluation and the representativeness of the open-source corpora.

assumptions (3)
  • domain assumption LLM evaluation scores for completeness, hallucination, readability, and usefulness are valid proxies for human judgment of documentation quality.
    Invoked in Section III-D2, where GPT-4o and Claude 3 are prompted to grade comments. No human evaluation is conducted, so the validity of the entire quality comparison rests on this assumption.
  • domain assumption SME partitions represent the correct ground truth for logical partition points in code.
    Used in Section III-D1 as the reference for precision and recall calculations. The number of SMEs is small (two MUMPS programmers and one ALC programmer), and their partition choices are treated as ground truth.
  • domain assumption The open-source corpora (Walmart ALC, VistA MUMPS) are representative of government legacy systems.
    Stated in Section III-B: 'We observed repeatable behavior of our methods, with government systems being similar to the Walmart and IRT code.' This generalization from open source to government systems is not backed by released data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Replace Humans During Code Chunking?." pith.science (2026). https://pith.science/paper/PXNXBW6S

@misc{pith2026250619897,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Replace Humans During Code Chunking?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXNXBW6S}},
  note         = {Machine review of arXiv:2506.19897}
}
read the original abstract

Large language models (LLMs) have become essential tools in computer science, especially for tasks involving code understanding and generation. However, existing work does not address many of the unique challenges presented by code written for government applications. In particular, government enterprise software is often written in legacy languages like MUMPS or assembly language code (ALC) and the overall token lengths of these systems exceed the context window size for current commercially available LLMs. Additionally, LLMs are primarily trained on modern software languages and have undergone limited testing with legacy languages, making their ability to understand legacy languages unknown and, hence, an area for empirical study. This paper examines the application of LLMs in the modernization of legacy government code written in ALC and MUMPS, addressing the challenges of input limitations. We investigate various code-chunking methods to optimize the generation of summary module comments for legacy code files, evaluating the impact of code-chunking methods on the quality of documentation produced by different LLMs, including GPT-4o, Claude 3 Sonnet, Mixtral, and Llama 3. Our results indicate that LLMs can select partition points closely aligned with human expert partitioning. We also find that chunking approaches have significant impact on downstream tasks such as documentation generation. LLM-created partitions produce comments that are up to 20% more factual and up to 10% more useful than when humans create partitions. Therefore, we conclude that LLMs can be used as suitable replacements for human partitioning of large codebases during LLM-aided modernization.

Figures

Figures reproduced from arXiv: 2506.19897 by the authors.

Figure 1
Figure 1. Comparing precision and recall of partitioning methods in relation to SME partitioning on each code corpus. Higher [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Completeness, usefulness, readability, and hallucination scores for GPT-4o, Claude Sonnet 3, Mixtral, and Llama 3 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Completeness, usefulness, readability, and hallucination scores for GPT-4o, Claude Sonnet 3, Mixtral, and Llama 3 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Hallucination scores for all generated comments for the MUMPS corpus based on partitioning method, partitioning [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Usefulness scores for all generated comments for the ALC corpus based on partitioning method, partitioning limit, and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 4 canonical work pages

  1. [1]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

  2. [2]

    Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts , 2022, pp. 1–7

    2022

  3. [3]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024

    2024

  4. [4]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” arXiv preprint arXiv:2406.00515, 2024

    arXiv 2024

  5. [5]

    Enhancing computer programming education with LLMs: A study on effective Prompt engineering for Python code generation,

    T. Wang, N. Zhou, and Z. Chen, “Enhancing computer programming education with LLMs: A study on effective Prompt engineering for Python code generation,” arXiv preprint arXiv:2407.05437 , Jul. 2024. Fig. 4: Hallucination scores for all generated comments for the MUMPS corpus based on partitioning method, partitioning limit, and the LLM that generated the c...

    arXiv 2024

  6. [6]

    EPiC: Cost-effective search-based prompt engineering of LLMs for code generation,

    H. Taherkhani, M. Sepindband, H. V . Pham, S. Wang, and H. Hemmati, “EPiC: Cost-effective search-based prompt engineering of LLMs for code generation,” arXiv preprint arXiv:2408.11198 , Aug. 2024

    arXiv 2024

  7. [7]

    Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering

    T. Zhang, Y . Yu, X. Mao, S. Wang, K. Yang, Y . Lu, Z. Zhang, and Y . Zhao, “Instruct or interact? Exploring and eliciting LLMs’ capability in code snippet adaptation through prompt engineering,” arXiv preprint arXiv:2411.15501, Nov. 2024

    work page Pith review arXiv 2024

  8. [8]

    Leveraging LLMs for legacy code modernization: Challenges and opportunities for LLM-generated documentation,

    C. Diggs, M. Doyle, A. Madan, S. Scott, E. Escamilla et al., “Leveraging LLMs for legacy code modernization: Challenges and opportunities for LLM-generated documentation,” arXiv preprint arXiv:2411.14971, Nov. 2024

    arXiv 2024

Show all 27 references
  1. [9]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,” Neurocomputing, vol. 568, p. 127063, 2024

    2024

  2. [10]

    Extending llms’ context window with 100 samples,

    Y . Zhang, J. Li, and P. Liu, “Extending llms’ context window with 100 samples,” arXiv preprint arXiv:2401.07004 , 2024

    2024 arXiv

  3. [11]

    Longbench: A bilingual, multitask benchmark for long context understanding,

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou et al. , “Longbench: A bilingual, multitask benchmark for long context understanding,” arXiv preprint arXiv:2308.14508, 2023

    2023 arXiv

  4. [12]

    Summary of a haystack: A challenge to long-context llms and rag systems,

    P. Laban, A. R. Fabbri, C. Xiong, and C.-S. Wu, “Summary of a haystack: A challenge to long-context llms and rag systems,” arXiv preprint arXiv:2407.01370, 2024

    2024 arXiv

  5. [13]

    From rags to rich parameters: Probing how language models utilize external knowl- edge over parametric information for factual queries,

    H. Wadhwa, R. Seetharaman, S. Aggarwal, R. Ghosh, S. Basu, S. Srini- vasan, W. Zhao, S. Chaudhari, and E. Aghazadeh, “From rags to rich parameters: Probing how language models utilize external knowl- edge over parametric information for factual queries,” arXiv preprint arXiv:2...

    2024 arXiv

  6. [14]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  7. [15]

    The chronicles of rag: The retriever, the chunk and the generator,

    P. Finardi, L. Avila, R. Castaldoni, P. Gengo, C. Larcher, M. Piau, P. Costa, and V . Carid´a, “The chronicles of rag: The retriever, the chunk and the generator,” arXiv preprint arXiv:2401.07883 , 2024. Fig. 5: Usefulness scores for all generated comments for the ALC corpus b...

    2024 arXiv

  8. [16]

    Is semantic chunking worth the computational cost?

    R. Qu, R. Tu, and F. Bao, “Is semantic chunking worth the computational cost?” arXiv preprint arXiv:2410.13070 , 2024

    2024 arXiv

  9. [17]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

    2023 arXiv

  10. [18]

    Meta- chunking: Learning efficient text segmentation via logical perception,

    J. Zhao, Z. Ji, P. Qi, S. Niu, B. Tang, F. Xiong, and Z. Li, “Meta- chunking: Learning efficient text segmentation via logical perception,” arXiv preprint arXiv:2410.12788 , 2024

    2024 arXiv

  11. [19]

    Lumberchunker: Long-form narrative document segmentation,

    A. V . Duarte, J. Marques, M. Grac ¸a, M. Freire, L. Li, and A. L. Oliveira, “Lumberchunker: Long-form narrative document segmentation,” arXiv preprint arXiv:2406.17526, 2024

    2024 arXiv

  12. [20]

    Grounding lan- guage model with chunking-free in-context retrieval,

    H. Qian, Z. Liu, K. Mao, Y . Zhou, and Z. Dou, “Grounding lan- guage model with chunking-free in-context retrieval,” arXiv preprint arXiv:2402.09760, 2024

    2024 arXiv

  13. [21]

    An empirical study on the code refactoring capability of large language models,

    J. Cordeiro, S. Noei, and Y . Zou, “An empirical study on the code refactoring capability of large language models,” arXiv preprint arXiv:2411.02320, 2024

    2024 arXiv

  14. [22]

    LLM$\times$MapReduce: Simplified long-sequence processing using large language models,

    Z. Zhou, C. Li, X. Chen, S. Wang, Y . Chao, Z. Li, H. Wang, R. An, Q. Shi, Z. Tan, X. Han, X. Shi, Z. Liu, and M. Sun, “LLM$\times$MapReduce: Simplified long-sequence processing using large language models,” arXiv preprint arXiv:2410.09342 , Oct. 2024

    arXiv

  15. [23]

    Llm-based and retrieval-augmented control code generation,

    H. Koziolek, S. Gr ¨uner, R. Hark, V . Ashiwal, S. Linsbauer, and N. Es- kandani, “Llm-based and retrieval-augmented control code generation,” in Proceedings of the 1st International Workshop on Large Language Models for Code , 2024, pp. 22–29

    2024

  16. [24]

    Python symbolic execution with llm-powered code generation,

    W. Wang, K. Liu, A. R. Chen, G. Li, Z. Jin, G. Huang, and L. Ma, “Python symbolic execution with llm-powered code generation,” arXiv preprint arXiv:2409.09271, 2024

    2024 arXiv

  17. [25]

    Towards translating real-world code with LLMs: A study of translating to Rust,

    H. F. Eniser, H. Zhang, C. David, M. Wang, M. Christakis, B. Paulsen, J. Dodds, and D. Kroening, “Towards translating real-world code with LLMs: A study of translating to Rust,” arXiv preprint arXiv:2405.11514, May 2024

    2024 arXiv

  18. [26]

    Bridging Eras: Transforming Fortran legacies into Python with the power of large language models,

    R. Pietrini, M. Paolanti, and E. Frontoni, “Bridging Eras: Transforming Fortran legacies into Python with the power of large language models,” in 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI) . IEEE, Apr. 2024

    2024

  19. [27]

    Batch prompting: Efficient inference with large language model apis,

    Z. Cheng, J. Kasai, and T. Yu, “Batch prompting: Efficient inference with large language model apis,” 2023. [Online]. Available: https://arxiv.org/abs/2301.08721 APPENDIX A. Partition Prompt Your purpose is to partition {SOURCE_LANGUAGE} code into self-contained logical blocks...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.