REVIEW 5 major objections 6 minor 2 cited by
Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper argues that a modernized form of literate programming, with Scheme-written zero-step/successor-step API descriptions, makes LLMs generate code more consistently across large projects.
desk verdict The ILP writing scheme is a plausible idea, but the paper's only detailed example embeds the answer in the prompt, and the RepoBench adaptation never produces a quantitative result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ILP document: a single literate-programming file in which every API is described by a zero-step (the base case or fundamental building block) and a successor-step (how one logical state moves to the next), written in Scheme-like notation with metadata such as complexity, stability, and examples, and arranged so the dependency graph among APIs forms an explicit directed acyclic graph with hyperlinks between chapters. The document does the work: because Scheme is uncommon in LLM training data, models cannot lean on memorized libraries and must reason from the two-step logic; the induced structure gives every generated function the same skeleton, which is why output becomes consistent across models.
What would settle it
Run the ILP prompt on the benchmark's original class-based Python repositories without rewriting them into Scheme modules, compute the benchmark's official pass rates, and compare against a plain-context baseline; if ILP does not improve the official scores, the central claim has no quantitative support. A second check: replace the Scheme descriptions in the toy cases with equally detailed Python descriptions; if the models then succeed as often, the claimed benefit of Scheme's scarcity is not the operative mechanism.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the format of the documentation, not the size of the model, can carry the burden of code-generation quality for large projects. When each API in a project is documented as a base case (zero-step) plus a transition (successor-step), written in Scheme and cross-linked so that a model can trace which APIs depend on which, the model is forced to follow the documented logic because Scheme is scarce in its training data. The paper reports that under this ILP guidance, a small local model produces take-right implementations in Python with the same edge-case handling as a much larger commercial model, while without the guidance implementations vary and omit error checks. It also reports that all toy Scheme functions tested with ILP guidance succeeded, while the same models without the documentation failed on several.
Load-bearing premise
The load-bearing premise is that the paper's adapted repository-level benchmark still measures repository-level code generation after class-based Python repositories are rewritten as Scheme-style modules and the evaluation criteria are swapped for functional-programming criteria; if that adaptation does not faithfully represent large-scale project code generation, the empirical evidence for the central claim collapses.
Editorial extensions
If this is right
- ILP-guided generation should reduce the spread of code quality across model sizes, so teams stuck with small local models get output comparable to large commercial ones.
- Developers can document an API before it exists, because the DAG links descriptions of not-yet-implemented functions; the LLM can generate implementations for all of them once prompted.
- Because the document, not the prompt, carries the specification, the same ILP document can generate code in different target languages from the same logic description.
- Adopting ILP should reduce retries and debugging: the two-step structure isolates the base case and the transition, so when code is wrong the error is localized to one documented step.
Reading between the lines
- The same two-step documentation trick should transfer to any deliberately unfamiliar notation; if ILP's benefit comes from Scheme's scarcity in training data, it should shrink as future models see more Scheme.
- A natural quantitative follow-up is to report official pass rates on both the Scheme-adapted and the unmodified original benchmark tasks, which would show how much of the ILP benefit is execution success rather than structural similarity.
- ILP documents could be synthesized automatically from an existing codebase's call graph and docstrings, removing the main adoption cost for legacy large projects.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Interoperable Literate Programming (ILP), a documentation format that combines natural language, Scheme-like descriptive snippets, and a zero-step/successor-step decomposition of API logic, and claims that prompting LLMs with ILP documents improves code generation for large-scale projects. It reports two evaluations: a pass/fail table for 12 toy Scheme list functions (Table 1) and a set of six Python implementations of take_right generated under ILP and non-ILP prompts (Sec. I.2). The latter is presented as a RepoBench result, although RepoBench was substantially adapted by converting class-based Python repositories into Scheme modules and replacing original metrics with functional-programming criteria (Sec. F.1). The paper concludes that ILP reduces variability, increases efficiency, and supports bulk API generation and complex interdependencies.
Significance. The idea of structuring literate-programming documents as explicit zero-step/successor-step specifications is a plausible and falsifiable prompt-engineering pattern that might help LLM-based code generation. The paper is clearly written about its proposed workflow, and it explicitly discusses limitations of alternative benchmarks (Sec. F.2). However, the empirical support is far too thin to substantiate the central claim about large-scale project development: Table 1 covers 12 simple functions with no repetitions or variance, and the only detailed example is a single small function whose ILP document contains the implementation structure. The paper does not ship machine-checked proofs, reproducible evaluation pipelines, or aggregate metrics; the reported code listings are illustrative. If the ILP approach were evaluated with rigorous, repository-scale metrics, it could be a useful contribution, but that evaluation is not in the current manuscript.
major comments (5)
- [Sec. 4.5, Sec. I.2, Abstract] The central claim that ILP 'enhances LLM-based code generation in large-scale project development' is not supported by the reported RepoBench experiment. The only result in Sec. I.2 is a set of Python code listings for the single function take_right, generated by six models, with no pass/fail counts, no execution results, no official RepoBench scores, no aggregate metrics, and no baseline statistics. A single toy example cannot demonstrate improvement on large-scale projects, and the abstract's assertion of 'enhancing ... large-scale project development' is therefore unsupported. The authors need to report correctness and other quantitative metrics over the actual RepoBench tasks, with and without ILP guidance.
- [Sec. F.1] The adaptation of RepoBench is not validated as a measure of repository-level code generation. The paper converts Python class repositories into Scheme modules, replaces object-oriented evaluation metrics with functional-programming criteria (pure functions, recursion patterns, immutability), and rewrites prompt templates. These changes alter the benchmark's task so thoroughly that results on the adapted benchmark cannot be assumed to transfer to the original RepoBench or to real-world large-scale Python development. The authors provide no evidence that the adapted metrics correlate with code correctness or with the original RepoBench scores. Because all empirical claims about scale rest on this adapted benchmark, its validity is load-bearing.
- [Sec. 3, Sec. I.2] The ILP document for take-right contains the solution structure: the zero-step says 'return the list if it's empty', the succ-step says 'otherwise, (drop lst 1) and (take-right lst (- n 1))', and the helper function drop is explicitly defined in the document. Consequently, the generated Python code is essentially a transliteration of the provided specification. This demonstrates that the models can follow detailed instructions, not that ILP helps with interdependencies, scale, or external-bias mitigation. To support the paper's claim, the authors need a head-to-head comparison on tasks where the ILP document describes the DAG and dependencies without embedding the step-by-step implementation.
- [Table 1, Sec. 4.2, contribution list] Table 1 reports pass/fail outcomes for 12 toy Scheme list functions without repetitions, error bars, or statistical tests. The paper's contributions claim that 'ILP reduces variability in code generation quality' and 'ILP improves code generation efficiency, achieving speed increases', but no variance measurements, repeated sampling, or timing data are reported anywhere. The claims about variability, efficiency, and reduced external bias therefore go beyond the evidence. These claims should be either removed or properly measured.
- [Sec. 4.4, Sec. 5] The claims of 'bulk API generation' and resolving 'complex interdependencies, including those involving components yet to be developed' are not evaluated. The inter-file inference example in Fig. 5 is a short Scheme description for an extended add function, and no experiment shows LLMs generating multiple interdependent APIs from an ILP document. The conclusion repeats these capabilities as demonstrated ('we showed that ILP enhances ... scalability'), but the manuscript contains no such demonstration.
minor comments (6)
- [Sec. B.1] The subsection heading 'Chanllenges of LLM-based code generation' contains a typo; it should be 'Challenges'.
- [Sec. B.2] In the example input, 'procedire' should be 'procedure'.
- [Title page] The affiliation 'John Hopkins University' should be 'Johns Hopkins University'.
- [Table 1] The table uses 'R.' and 'D.' but does not report a GPT-4 (R.) column, so the reader cannot see the R7RS-only result for GPT-4; the caption or table should be made consistent.
- [Sec. 1] Mogan and Goldfish Scheme are mentioned in the introduction before being defined; a one-sentence definition at first mention would help.
- [Sec. I.2] The code listings are not labeled with which model generated which 'with ILP' and 'without ILP' outputs in a parallel structure; adding a small table of correctness (e.g., edge-case behavior) for each listing would make the comparison clearer.
Circularity Check
Central ILP evaluation is partly circular: the adapted RepoBench metric rewards adherence to the Step-wise method, and the flagship take-right example supplies the solution structure in the prompt; external R7RS pass/fail results provide only a partial anchor.
-
self definitional
[Appendix F.1 (RepoBench Adaptation for Scheme Language)]
"Second, we redefine RepoBench’s evaluation metrics to better capture functional programming characteristics. ... We replace these with metrics evaluating function composition, state management via immutable data structures, and recursive correctness. ... We also enhance the evaluation framework to assess literate programming by adding metrics for documentation completeness, integration with code, and adherence to Step-wise methods specified in the prompt."
The ILP intervention is precisely the Step-wise (zero-step/succ-step) documentation method. By redefining the evaluation metric to include 'adherence to Step-wise methods specified in the prompt', the benchmark's success criterion coincides with the intervention's own prescription. The paper then claims (Sec. 5) that experiments on this adapted RepoBench show ILP 'enhances code generation quality, efficiency, and scalability' without reporting official RepoBench scores. The measured improvement is therefore built into the metric rather than demonstrated against an independent standard of large-scale code generation.
-
other
[Sec. 3 (ILP documentation design) and Sec. I.2 (RepoBench Python code generation)]
"### Helper Function: `drop` The helper function `drop` removes the first *n* elements from a list. ... ;; Base (zero-step): ;; if n <= 0 or lst is empty, return lst as is ;; Succ-step: ;; otherwise, (drop lst 1) and (take-right lst (- n 1))"
The ILP document supplied to the models contains the base case, the recursive/successor structure, and the helper function drop. The Sec. I.2 'with ILP' Python outputs (e.g., ChatGPT-4's take_right with 'if not flist or i >= len(flist): return flist' and 'return drop(flist, skip_count)') are near transliterations of that specification. The comparison against 'without ILP' outputs therefore largely measures whether the model can copy a detailed recipe; it does not independently test whether ILP enables models to handle complex, large-scale interdependencies. This is a fitted-input-called-prediction pattern: the prompt is constructed to contain the answer, and the generated code is then presented as evidence of ILP's benefit.
full rationale
The paper's central claim is empirical, and there is a genuine controlled comparison (with vs without ILP on the same function), so this is not an instance of the entire derivation collapsing. The R7RS toy-case success/failure table (Table 1) is an independent functional check. However, the large-scale claim rests on (i) a RepoBench adaptation whose evaluation criteria incorporate 'adherence to Step-wise methods specified in the prompt', and (ii) a single Python example (take-right) whose ILP document literally contains the zero-step, succ-step, and helper drop that the generated code reproduces. No official RepoBench scores or pass rates are reported in Sec. I.2. Thus the portion of the evidence that is supposed to show ILP's advantage at repository scale reduces, at least in part, to the model following the solution-bearing specification and to a metric that is aligned with the intervention. Self-citations (Goldfish Scheme, Mogan) are present but not load-bearing for this circularity; they do not raise the score further. Overall score 6 reflects partial circularity with an independent toy-case anchor.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper Every API's logic can be decomposed into a base (zero-step) and successor-step structure without loss of generality.
- domain assumption Scheme is sufficiently rare in LLM training data that models must rely on provided documentation rather than pretrained patterns.
- domain assumption LLMs can recursively traverse and retrieve dependencies across ILP document chapters when generating cross-file code.
- domain assumption RepoBench remains a valid benchmark after restructuring repositories, redefining evaluation metrics, and translating comments to Scheme.
Cite this review
Pith. "Pith review of Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects." pith.science (2026). https://pith.science/paper/6FQWOMAR
@misc{pith2026250217441,
author = {Pith},
title = {Pith review of: Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects},
year = {2026},
howpublished = {\url{https://pith.science/paper/6FQWOMAR}},
note = {Machine review of arXiv:2502.17441}
}
read the original abstract
Large Language Models (LLMs) have helped programmers increase efficiency through code generation, comprehension, and repair. However, their application to large-scale projects remains challenging due to complex interdependencies and the extensive size of modern codebases. Although Knuth's concept of Literate Programming (LP) combines code and natural language to convey logic and intent, its potential for enhancing relationships in large projects has not been fully explored. In this study, we introduce the idea of Interoperable LP (ILP), which leverages literate programming principles to enhance the development of both small-scale documents and large-scale projects with LLMs. We investigate how LLMs perform under ILP-style instructions for both document-oriented tasks and entire projects. Recognizing that many researchers rely on well-structured templates to guide LLMs, we propose a concise prompt engineering method to write LP documents so LLMs can better be involved in code generation. We also examine the capacity of various LLMs to generate Scheme and Python code on the RepoBench benchmark, illustrating the advantages of our approach. Our findings indicate that ILP with LLMs can enhance LLM-based code generation in large-scale project development.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 2 Pith papers
-
Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation
A physics-aware two-stage SFT+GRPO training method with period/AST/sandbox rewards raises small open LLMs from ~0-2% to ~68-77% on a strict OpenSeesPy building-modeling benchmark (BMEval).
-
A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis
An LLM agent that reframes beam analysis as OpenSeesPy code generation reaches over 99 percent reliability on a small benchmark, but chiefly because the prompt contains a near-identical solved example.
Reference graph
Works this paper leans on
-
[1]
Zero-step definition: We start by specifying abase caseor fundamental building block. In recursive functions, for example, this might be the condition under which the recursion terminates. By designating this zero-step, we give the LLM a clear anchor point for the broader logic
- [2]
-
[3]
This structure helps the LLM associate specific logic with the corresponding implementation
Code chunks per step: Each step’s code is presented in a self-contained chunk, accompanied by an explanation of its function (e.g., base case, recursive step, edge cases). This structure helps the LLM associate specific logic with the corresponding implementation. 20 Renaissance of Literate Programming in the Era of LLMs
-
[4]
Local Verification and Iteration: After a chunk is introduced, we encourage local testing (for example, in Mogan’s REPL) to confirm correctness. If an error occurs, developers can isolate it to that chunk and provide the LLM with focused feedback, minimizing the spread of mistakes
-
[5]
Create a class that implements a priority queue with methods for insertion and deletion
IncrementalComplexity: Onceabaseversionpassesverification, weaddoptionalfeaturesoroptimizationsinsmaller, separate steps. Each new layer is validated independently, preventing confusion that can occur when many changes happen at once. For example, in Scheme • The Logic of computation in Scheme: All logic is defined in Scheme for precise, induction-friendl...
-
[7]
Project-level context understanding
-
[8]
Documentation generation assessment
-
[9]
Cross-file dependency handling
Show all 19 references
-
[10]
Step-wise method:
Integration with existing codebase patterns RepoBench’s repositories also provide real-world examples of API evolution, which aligns with our goal of supporting API development through literate programming. Each repository includes multiple versions of APIs, showing how they e...
-
[11]
LLMs can effectively generate code within specific stylistic constraints when properly guided
-
[12]
transfer learning
ILP Documentation structure significantly enhanced LLMs’ ability to adhere to prescribed patterns. G.1 Machine details Our experiments are conducted on two types of LLMs: local open-source LLMs and remote commercial LLMs. Remote commercial LLMs For remote commercial LLMs, quer...
-
[13]
First, it enables developers to leverage Scheme’s Step-wise methods and functional purity for critical algorithmic components while maintaining compatibility with existing systems
Domain-specific validation" (let ((inner-validate (lambda (data acc) ;; Implementation pending ))) (inner-validate data '()))) This hybrid structure serves multiple purposes. First, it enables developers to leverage Scheme’s Step-wise methods and functional purity for critical...
-
[14]
AlgorithmPrototyping: TeamscanexpresscomplexalgorithmsinSchemewithmathematicalrigorbeforeimplementing them in production languages
-
[15]
Interface Design: Developers can maintain existing interfaces while evolving core implementations
-
[16]
"" @requires: Implementation in core_processor.scm @ensures: Output satisfies sorting invariants
DocumentationFirst: Teamscanuseliterateprogrammingtofullyspecifybehaviorbeforecommittingtoimplementation details The effectiveness of this hybrid approach becomes evident in testing scenarios. When generating code, language models must understand both: # Python interface speci...
-
[17]
Order preservation for equivalent elements
-
[18]
Linear time complexity
-
[19]
"" Returns the sublist of lst that remains after dropping n elements from the front. If n is 0 or less, or if lst is empty, returns lst unchanged
Space complexity O(n)" (implementation ...)) This structure reflects real development workflows where teams might: 1. Design APIs in familiar languages 2. Document core algorithms in Scheme for mathematical clarity 3. Implement and verify critical components functionally 4. Br...
2024
-
[1481]
gradient descent
IEEE, 2023. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. Gamma, E. Design patterns: elements of reu...
2020 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.