{"id":"fdc84365-c137-40aa-a318-c61f11ac57e7","arxiv_id":"2502.17441","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A documentation format based on literate programming is claimed to improve LLM code generation, but the experiments show only toy-level success/failure results and a single RepoBench function.","lead":"This paper introduces Interoperable Literate Programming, a documentation format meant to help large language models write code for large projects. The authors claim it improves code consistency and reliability, but their experiments are mostly small toy cases and a single example function.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated benchmark and a single toy example whose ILP document embeds the implementation; no quantitative repository-scale evidence is reported.","rationale":"The reader's REJECT verdict is correct in direction, and the adapted-benchmark problem is real, but I locate the most load-bearing failure more specifically. For the abstract's claim to hold, two things are needed: (a) an evaluation that represents repository-scale generation with dependencies, and (b) a comparison showing ILP documents, not just solution hints, cause improvement. The paper supplies neither. Sec. F.1 explicitly changes the benchmark: object-oriented repos become Scheme modules and metrics change to functional-programming criteria, so even if scores existed, they would not be RepoBench scores. No scores are given in Sec. I.2; the appendix prints code. The toy Table 1 is not quantitative (no run counts, no variance) and is not repository-scale. The take-right document is the clearest internal problem: the 'succ-step' comment literally describes the recursive call to the function and the document defines `drop`; the LLM's job is to translate Scheme into Python. Thus the 'ILP reduces variability' claim is explained by the document carrying the algorithm. This is a circularity concern routed through correctness risk: the demonstration is consistent with the hypothesis but also with the trivial hypothesis that LLMs transliterate given code. A clean test is the ablation above. That said, the paper makes a useful proposal (structured LP docs, Mogan, Goldfish) and does not overclaim in the discussion; my objection is to the empirical support, not the idea. Because the central evidence gap is large and a re-run with solution-free documents and official RepoBench metrics could reverse the finding, I concur with REJECT.","tokens_in":27361,"tokens_out":3840,"duration_ms":36348,"concrete_test":"Run a held-out, solution-free ablation on RepoBench's original Python evaluation. (1) Build ILP documents from API signatures and dependency DAGs only—zero-step/succ-step at specification level, without succ-step implementation comments such as '(drop lst 1) and (take-right lst (- n 1))' and without defining helper `drop` in the document. (2) Generate Python functions with the same models (GPT-4, Claude, Llama 3.1 2B, Qwen) on at least 50 repository functions. (3) Score with the original RepoBench test harness and metrics, not the Sec. F.1 redefined functional criteria. If the ILP-vs-no-ILP gap on official pass@k collapses once the implementation is removed from the document, the paper's reported advantage is an artifact of solution leakage rather than of ILP's DAG/step-wise structure; if official scores are never reported, the large-scale claim remains unevidenced.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that ILP with LLMs enhances LLM-based code generation in large-scale project development (Abstract)—requires evidence on real, interdependent, repository-scale code. The paper's only detailed generation result is Sec. I.2, which shows Python implementations of `take_right` from six models, without pass rates, official RepoBench scores, or baseline statistics. Table 1 is a yes/no table over toy Scheme list functions. The one RepoBench adaptation (Sec. F.1) redraws the evaluation: Python class repositories are restructured as Scheme modules, object-oriented metrics are replaced by functional-programming criteria, and prompts are rewritten; Sec. I.2 then reports only code listings. The load-bearing assumption that this adapted benchmark is a valid test of large-scale code generation is therefore not met by evidence in the paper. Further, the ILP document in Sec. 3 for `take-right` contains the solution structure: the succ-step comment states 'otherwise, (drop lst 1) and (take-right lst (- n 1))', and the helper `drop` is defined in the document. Generated Python code is essentially a transliteration. That demonstrates prompt-following, not scale or interdependency handling. Without a head-to-head, solution-free ablation on official RepoBench tasks, the central claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Interoperable Literate Programming (ILP), a documentation format that combines natural language, Scheme-like descriptive snippets, and a zero-step/successor-step decomposition of API logic, and claims that prompting LLMs with ILP documents improves code generation for large-scale projects. It reports two evaluations: a pass/fail table for 12 toy Scheme list functions (Table 1) and a set of six Python implementations of take_right generated under ILP and non-ILP prompts (Sec. I.2). The latter is presented as a RepoBench result, although RepoBench was substantially adapted by converting class-based Python repositories into Scheme modules and replacing original metrics with functional-programming criteria (Sec. F.1). The paper concludes that ILP reduces variability, increases efficiency, and supports bulk API generation and complex interdependencies.","tokens_in":27587,"tokens_out":5008,"duration_ms":47519,"significance":"The idea of structuring literate-programming documents as explicit zero-step/successor-step specifications is a plausible and falsifiable prompt-engineering pattern that might help LLM-based code generation. The paper is clearly written about its proposed workflow, and it explicitly discusses limitations of alternative benchmarks (Sec. F.2). However, the empirical support is far too thin to substantiate the central claim about large-scale project development: Table 1 covers 12 simple functions with no repetitions or variance, and the only detailed example is a single small function whose ILP document contains the implementation structure. The paper does not ship machine-checked proofs, reproducible evaluation pipelines, or aggregate metrics; the reported code listings are illustrative. If the ILP approach were evaluated with rigorous, repository-scale metrics, it could be a useful contribution, but that evaluation is not in the current manuscript.","major_comments":[{"comment":"The central claim that ILP 'enhances LLM-based code generation in large-scale project development' is not supported by the reported RepoBench experiment. The only result in Sec. I.2 is a set of Python code listings for the single function take_right, generated by six models, with no pass/fail counts, no execution results, no official RepoBench scores, no aggregate metrics, and no baseline statistics. A single toy example cannot demonstrate improvement on large-scale projects, and the abstract's assertion of 'enhancing ... large-scale project development' is therefore unsupported. The authors need to report correctness and other quantitative metrics over the actual RepoBench tasks, with and without ILP guidance.","section":"Sec. 4.5, Sec. I.2, Abstract"},{"comment":"The adaptation of RepoBench is not validated as a measure of repository-level code generation. The paper converts Python class repositories into Scheme modules, replaces object-oriented evaluation metrics with functional-programming criteria (pure functions, recursion patterns, immutability), and rewrites prompt templates. These changes alter the benchmark's task so thoroughly that results on the adapted benchmark cannot be assumed to transfer to the original RepoBench or to real-world large-scale Python development. The authors provide no evidence that the adapted metrics correlate with code correctness or with the original RepoBench scores. Because all empirical claims about scale rest on this adapted benchmark, its validity is load-bearing.","section":"Sec. F.1"},{"comment":"The ILP document for take-right contains the solution structure: the zero-step says 'return the list if it's empty', the succ-step says 'otherwise, (drop lst 1) and (take-right lst (- n 1))', and the helper function drop is explicitly defined in the document. Consequently, the generated Python code is essentially a transliteration of the provided specification. This demonstrates that the models can follow detailed instructions, not that ILP helps with interdependencies, scale, or external-bias mitigation. To support the paper's claim, the authors need a head-to-head comparison on tasks where the ILP document describes the DAG and dependencies without embedding the step-by-step implementation.","section":"Sec. 3, Sec. I.2"},{"comment":"Table 1 reports pass/fail outcomes for 12 toy Scheme list functions without repetitions, error bars, or statistical tests. The paper's contributions claim that 'ILP reduces variability in code generation quality' and 'ILP improves code generation efficiency, achieving speed increases', but no variance measurements, repeated sampling, or timing data are reported anywhere. The claims about variability, efficiency, and reduced external bias therefore go beyond the evidence. These claims should be either removed or properly measured.","section":"Table 1, Sec. 4.2, contribution list"},{"comment":"The claims of 'bulk API generation' and resolving 'complex interdependencies, including those involving components yet to be developed' are not evaluated. The inter-file inference example in Fig. 5 is a short Scheme description for an extended add function, and no experiment shows LLMs generating multiple interdependent APIs from an ILP document. The conclusion repeats these capabilities as demonstrated ('we showed that ILP enhances ... scalability'), but the manuscript contains no such demonstration.","section":"Sec. 4.4, Sec. 5"}],"minor_comments":[{"comment":"The subsection heading 'Chanllenges of LLM-based code generation' contains a typo; it should be 'Challenges'.","section":"Sec. B.1"},{"comment":"In the example input, 'procedire' should be 'procedure'.","section":"Sec. B.2"},{"comment":"The affiliation 'John Hopkins University' should be 'Johns Hopkins University'.","section":"Title page"},{"comment":"The table uses 'R.' and 'D.' but does not report a GPT-4 (R.) column, so the reader cannot see the R7RS-only result for GPT-4; the caption or table should be made consistent.","section":"Table 1"},{"comment":"Mogan and Goldfish Scheme are mentioned in the introduction before being defined; a one-sentence definition at first mention would help.","section":"Sec. 1"},{"comment":"The code listings are not labeled with which model generated which 'with ILP' and 'without ILP' outputs in a parallel structure; adding a small table of correctness (e.g., edge-case behavior) for each listing would make the comparison clearer.","section":"Sec. I.2"}],"recommendation":"reject","confidential_remarks":"The paper would be more appropriately positioned as a position or tooling paper, but as a research claim about large-scale code generation it falls short. The most serious issue is that the only detailed example is confounded by the ILP document containing the implementation, and the benchmark adaptation has not been validated. Even a strong revision would need new, large-scale experiments, which goes well beyond local fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe genuinely new piece here is the ILP document format: zero-step/succ-step decomposition written in Scheme-ish pseudocode, plus a define-with-docs macro for DAG-style annotations. That is a real extension of Shi et al.'s natural-language outlines, and the two tools, Mogan and Goldfish Scheme, are plausibly useful. The paper also deserves credit for being clear about what LP is and what it adds, and the related-work coverage is adequate.\n\nThe problem is the evidence. The central claim—ILP enhances LLM code generation in large-scale projects—is supported only by Table 1 (pass/fail for 12 toy Scheme functions, no repetitions, no error bars) and by the take_right listings in Sec. I.2. And take_right is the worst possible exemplar: the ILP document literally contains the succ-step \"otherwise, (drop lst 1) and (take-right lst (- n 1))\" and defines drop. The generated Python is a transliteration. That shows prompt-following, not repository-scale interdependency handling.\n\nThe RepoBench section makes this worse. The authors restructure Python classes into Scheme modules, replace OO metrics with functional-programming criteria, and rewrite prompts. Then they report one function's code listing, with no test execution, no official score, no baseline numbers. By the time the benchmark is adapted, calling it RepoBench is mostly aspirational. Claims about speed, variability reduction, and bulk API generation have no quantitative backing anywhere, and no evaluation harness is released. This is a load-bearing gap, not a cosmetic one.\n\nI don't think the idea should be discarded. The ILP recipe is coherent and the toy result is consistent with it. But the paper currently reads like a tool/position paper dressed up as an empirical study. I'd like to see the authors either reframe it as such or run a real evaluation: official RepoBench (or SWE-bench) tasks, ILP documents that do not contain the solution, pass@k with error bars, and an ablation against plain CoT or Shi-style outlines.\n\nRecommendation: worth a serious referee, but not in current form. Send it out mainly so the reviewers can push for that evidence.","headline":"The ILP writing scheme is a plausible idea, but the paper's only detailed example embeds the answer in the prompt, and the RepoBench adaptation never produces a quantitative result.","tokens_in":28142,"tokens_out":2885,"would_cite":false,"duration_ms":28438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a modernized form of literate programming, with Scheme-written zero-step/successor-step API descriptions, makes LLMs generate code more consistently across large projects.","keywords":["literate programming","interoperable literate programming","LLM-based code generation","prompt engineering","Scheme","functional programming","repository-level code generation","directed acyclic graph"],"falsifier":"Run the ILP prompt on the benchmark's original class-based Python repositories without rewriting them into Scheme modules, compute the benchmark's official pass rates, and compare against a plain-context baseline; if ILP does not improve the official scores, the central claim has no quantitative support. A second check: replace the Scheme descriptions in the toy cases with equally detailed Python descriptions; if the models then succeed as often, the claimed benefit of Scheme's scarcity is not the operative mechanism.","tokens_in":27146,"feed_emoji":"📜","tokens_out":7648,"duration_ms":64100,"temperature":0.7,"pith_summary":"The paper claims that the decades-old idea of literate programming — writing code and prose together as one narrative — can be resurrected and extended to solve a current problem: LLMs generating code for large projects. The extension, called Interoperable Literate Programming (ILP), describes each API's logic as a two-step induction-style pattern, written in Scheme, with the dependency graph among APIs made explicit inside a single document. According to the paper, an LLM that is told to rely only on such a document produces more consistent code, handles cross-file dependencies including not-yet-written APIs, and shows less variability between large and small models. A reader should care because today's failure mode in large-project code generation is precisely that models fall back on their own training knowledge instead of a project's internal APIs, and ILP is offered as a documentation standard that removes that failure.","feed_headline":"Literate programming, rebooted, steadies LLM code generation","feed_subtitle":"Scheme-based zero-step/successor-step documents cut output variability and help small models match large ones.","key_machinery":"The central object is the ILP document: a single literate-programming file in which every API is described by a zero-step (the base case or fundamental building block) and a successor-step (how one logical state moves to the next), written in Scheme-like notation with metadata such as complexity, stability, and examples, and arranged so the dependency graph among APIs forms an explicit directed acyclic graph with hyperlinks between chapters. The document does the work: because Scheme is uncommon in LLM training data, models cannot lean on memorized libraries and must reason from the two-step logic; the induced structure gives every generated function the same skeleton, which is why output becomes consistent across models.","core_discovery":"On its own terms, the paper's discovery is that the format of the documentation, not the size of the model, can carry the burden of code-generation quality for large projects. When each API in a project is documented as a base case (zero-step) plus a transition (successor-step), written in Scheme and cross-linked so that a model can trace which APIs depend on which, the model is forced to follow the documented logic because Scheme is scarce in its training data. The paper reports that under this ILP guidance, a small local model produces take-right implementations in Python with the same edge-case handling as a much larger commercial model, while without the guidance implementations vary and omit error checks. It also reports that all toy Scheme functions tested with ILP guidance succeeded, while the same models without the documentation failed on several.","pith_inferences":["The same two-step documentation trick should transfer to any deliberately unfamiliar notation; if ILP's benefit comes from Scheme's scarcity in training data, it should shrink as future models see more Scheme.","A natural quantitative follow-up is to report official pass rates on both the Scheme-adapted and the unmodified original benchmark tasks, which would show how much of the ILP benefit is execution success rather than structural similarity.","ILP documents could be synthesized automatically from an existing codebase's call graph and docstrings, removing the main adoption cost for legacy large projects."],"forward_implications":["ILP-guided generation should reduce the spread of code quality across model sizes, so teams stuck with small local models get output comparable to large commercial ones.","Developers can document an API before it exists, because the DAG links descriptions of not-yet-implemented functions; the LLM can generate implementations for all of them once prompted.","Because the document, not the prompt, carries the specification, the same ILP document can generate code in different target languages from the same logic description.","Adopting ILP should reduce retries and debugging: the two-step structure isolates the base case and the transition, so when code is wrong the error is localized to one documented step."],"supporting_citations":[{"why":"supplies the literate programming principles (flexible code organization, tangling, weaving) that ILP extends.","marker":"(Knuth, 1984)"},{"why":"supplies the repository-level benchmark that the paper adapts for its evaluation.","marker":"(Liu et al., 2023)"},{"why":"supplies chain-of-thought prompting, the stepwise reasoning pattern the ILP documentation adopts.","marker":"(Wei et al., 2022)"},{"why":"introduces Scheme, the language used for ILP descriptions.","marker":"(Sussman & Steele Jr, 1975)"},{"why":"supplies Goldfish Scheme, the interpreter with a Python-like standard library used in the ILP toolchain.","marker":"(Shen et al., 2024)"},{"why":"documents the accuracy decline and interdependency failures in LLM code generation that ILP targets.","marker":"(Dou et al., 2024)"},{"why":"provides the natural-language-outlines contrast, which the paper argues lacks the project-wide export and structure of ILP.","marker":"(Shi et al., 2024)"}],"fun_headline_variants":["Scheme-based docs steady LLM code generation in large projects","Literate programming: small models match large ones with Scheme docs","Scheme docs reduce LLM output variability in large codebases","Cross-linked Scheme docs make small LLMs competitive"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper's adapted repository-level benchmark still measures repository-level code generation after class-based Python repositories are rewritten as Scheme-style modules and the evaluation criteria are swapped for functional-programming criteria; if that adaptation does not faithfully represent large-scale project code generation, the empirical evidence for the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Scheme-based docs steady LLM code generation in large projects","Literate programming: small models match large ones with Scheme docs","Scheme docs reduce LLM output variability in large codebases","Cross-linked Scheme docs make small LLMs competitive"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3547,"prompt_tokens":917,"completion_tokens":2630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2563}},"tokens_in":533,"tokens_out":2630,"duration_ms":17509,"temperature":1.0,"reasoning_tokens":2563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:21:42.461845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ILP prompt on the benchmark's original class-based Python repositories without rewriting them into Scheme modules, compute the benchmark's official pass rates, and compare against a plain-context baseline; if ILP does not improve the official scores, the central claim has no quantitative support. A second check: replace the Scheme descriptions in the toy cases with equally detailed Python descriptions; if the models then succeed as often, the claimed benefit of Scheme's scarcity is not the operative mechanism.","supporting_citations":[],"review_version":1}