{"id":"9030f3dc-9688-4cfa-bbe3-110f62e1c4b4","arxiv_id":"2509.17455","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ICRAG, a retrieval-augmented iterative code-generation framework, improves accuracy on 13 NLP benchmarks by compiling questions into Python programs and executing them, though gains are partly inflated by in-distribution retrieval.","lead":"A new system, ICRAG, turns benchmark questions into Python programs, retrieves external knowledge, and iteratively rewrites the code until it runs, reporting large accuracy gains on 13 benchmarks. A reader might care because it suggests a practical path to transparent, inspectable LLM reasoning, but the gains are weakened by retrieval from the same datasets being tested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transductive R2 leakage: k-fold retrieval pool includes labeled test-distribution analogs, so headline gains may not reflect held-out codification.","rationale":"The most load-bearing condition for the central empirical claim is that R2's k-fold construction prevents label leakage. Reading §3.2 and §4.1 carefully, the retrieval pool for a test instance contains 80% of the benchmark with answers embedded in code snippets. The authors' stated safeguard only rules out direct memorization of the exact instance, not analogical inference from similar labeled instances. This is a transductive evaluation setup; real-world deployment would not have a labeled pool from the target task. The paper's ablations (RAG_NL, IRCoT, RAG_Code) all consume the same R2, so they cannot isolate this effect; the comparison to Direct/CoT/CoC, which drives the 32.7% headline, is the one that is confounded. The proposed train-split R2 test would settle the concern: if gains persist, the claim is robust; if they collapse, the paper's conclusion is overstated. The reader's verdict of CONDITIONAL (medium correctness risk) is exactly right; I do not see a reason to accept or reject outright. I therefore keep the verdict unchanged. I note also the title/abstract mismatch and the tautological correctness metric for nongeneralizable tasks, but these are secondary to the leakage issue.","tokens_in":35280,"tokens_out":7898,"duration_ms":69530,"concrete_test":"Rebuild R2 exclusively from each dataset's official training split (excluding all test instances) and rerun Table 3 under identical settings. If the 13-dataset average accuracy drops by more than 5 absolute points (e.g., from 0.73 to below ~0.68) or any dataset falls below its CoC row, the headline gains are attributable to transductive leakage from the k-fold R2. Also report an R1-only condition to isolate the external-knowledge contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 constructs R2 as code snippets s=LLM(d) of solved examples drawn from the same benchmark via k-fold (k=5, §4.1). Each snippet embeds the gold answer via Prompt A.1; Figure 2's retrieved snippet contains 'is_debt_collector = not is_original_creditor' and the final code prints a label. For a given test instance, the retriever returns top-3 snippets from the other 4/5 of the dataset, i.e., labeled solved instances from the same distribution. The k-fold scheme only prevents exact answer lookup; it does not prevent transductive label leakage: the LLM can infer the held-out answer by analogy to retrieved labeled snippets (nearest-neighbor label transfer). This inflates ICRAG's accuracy relative to Direct/CoT/CoC, which see no labeled in-distribution examples. The ablations RAG_NL and IRCoT also use R2, so they share the leakage and do not control for it. The paper never reports ICRAG with R1 only or with R2 built from a proper training split. Consequently, the central claim that ICRAG 'consistently exceeds text-only reasoning and one-shot code execution' may be an artifact of the transductive retrieval pool rather than of iterative code refinement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ICRAG, a system that converts natural-language benchmark instances into Python programs via an LLM, augments each generation with retrieval from two sources (domain corpora R1 and a dataset-derived pool R2), and iteratively refines the program by executing it and feeding execution errors and retrieved snippets back to the LLM. The authors evaluate ICRAG on 13 benchmarks spanning math, causal reasoning, legal judgment, biomedical QA, and Big-Bench-Hard/Extra-Hard, reporting an average 32.7% absolute accuracy improvement over text-only and one-shot code-execution baselines. They also introduce the notion of \"computables\" and analyze the generated code's complexity, AST structure, and t-SNE embedding to argue that natural-language tasks yield a distinctive class of programs.","tokens_in":35498,"tokens_out":4538,"duration_ms":42857,"significance":"If the central accuracy claim held up, this would be a useful empirical contribution: it would show that iterative retrieval-augmented code generation with execution feedback generalizes codification from math puzzles to legal, medical, and causal-reasoning tasks, and that the resulting programs provide inspectable evidence of reasoning. The paper has notable strengths: it covers a broad benchmark suite, reports controlled external-knowledge experiments (E1-E3) that separately vary in-domain pool size and GitHub-code injection, provides the exact prompt templates and FAISS configuration in appendices, and includes cross-model robustness checks with Claude, Gemini, DeepSeek, GPT-3.5, and Llama. The AST-profile and complexity analyses of generated code are novel and could be of independent interest. However, the headline comparison against no-retrieval baselines is undermined by the transductive construction of R2, and the \"correctness\" metric for nongeneralizable tasks is a tautology as stated.","major_comments":[{"comment":"The retrieval pool R2 is built from a (k-1)/k portion of each benchmark itself, and each snippet s = LLM(d) is generated via Prompt A.1, which explicitly includes the gold answer and asks for a program that \"returns the final answer.\" The k=5 split only excludes the exact test instance; it does not prevent transductive label leakage. For the classification-style datasets (CAIL with 150 classes, ECHR, CAP, PubMedQA, Health-Claim, BioASQ, and many BBH/BBEH tasks), the top-3 retrieved snippets are labeled in-distribution analogs, so the LLM can infer the held-out answer by nearest-neighbor analogy or by matching the retrieved code's printed label. Direct, CoT, and CoC see no such labeled examples, so the reported +32.7% average gain and the claim that ICRAG \"consistently exceeds text-only reasoning and one-shot code execution\" may reflect an unfair advantage rather than the method itself. The ablations RAG_NL, RAG_Code, and IRCoT share R2 and therefore do not control for this. Please add an ablation with R1 only, or rebuild R2 from a proper training split, or otherwise demonstrate that the gains over no-retrieval baselines persist when the retrieval pool contains no in-distribution labeled answers.","section":"§3.2, §4.1, Table 1"},{"comment":"The paper states that for nongeneralizable tasks \"the correctness is by definition the same as its one-time execution success on that instance\" and Table 4 reports correctness values that are merely the original execution accuracies with small unexplained deviations. This is not a test of generalization; it is a restatement of the accuracy numbers. The claim in Section 5.3 that correctness \"evaluates whether a generated program consistently produces correct outputs when applied to new valid inputs\" is therefore not supported for the legal and health-claim datasets. Either remove the correctness framing for nongeneralizable tasks, or design a genuinely out-of-sample evaluation (e.g., human-validated input perturbations that change case facts while preserving the applicable legal or medical rule) and report how the perturbations were generated and verified.","section":"§5.3.3, Table 4"},{"comment":"The ProofNet evaluation is inconsistent and under-validated. Section 4.1 says the OpenAI evaluation platform applies \"the GPT-o3 model\" to grade proofs, while Section 4.2 says \"a GPT-3-based grader\" was used. No evidence is provided that this automatic grader is reliable: there is no human validation sample, no agreement statistic, and no comparison with the benchmark's official protocol. Because ProofNet is one of the 13 datasets and shows one of the largest ICRAG improvements (0.74 to 0.98), the grader's reliability is load-bearing for that row. Please clarify which model was actually used, and provide validation of the grader on a held-out sample (e.g., human-judged correctness or agreement with a formal proof checker).","section":"§4.1, §4.2, Table 3"}],"minor_comments":[{"comment":"Direct, CoT, and CoC columns report point estimates without variance, while the RAG-based methods include standard deviations; please state the number of runs/seeds for all methods and report variances uniformly.","section":"Table 3"},{"comment":"The phrase \"GPT-3-based grader\" in Section 4.2 contradicts the \"GPT-o3\" statement in Section 4.1; this should be reconciled and the grader version stated precisely.","section":"Section 4.2"},{"comment":"The appendix states that the illustrative examples \"are not actual dataset samples,\" but several examples are described as reflecting the dataset styles; please mark clearly which examples are synthetic and which, if any, are real dataset instances, and provide the exact prompts used for non-math domains (Appendix A says the templates are for \"all Math experiments\" but the method is applied to legal and medical tasks).","section":"Appendix E"},{"comment":"The unified panel for E1/E2/E3 is difficult to read: the x-axis is shared across two different meanings of r, and the E3 points at x=75% are confusing since E3 uses only GitHub code; please split the plots or clarify the axes and point labels.","section":"Figure 3"},{"comment":"The t-SNE embedding lacks methodological details such as perplexity, number of iterations, and distance metric; the informal axis labels \"More Textual\" and \"More Complex Reasoning\" should be justified or replaced with concrete feature directions.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the empirical scope is impressive, but the transductive R2 construction is a serious confound that must be addressed before the headline claims can be accepted. I would not reject outright: the authors have the ablative machinery (R1-only, train-split R2, or baseline-with-R2) to repair this within the manuscript's scope. The ProofNet grader inconsistency and the tautological correctness metric also need fixing. If the authors can show that ICRAG's advantage over no-RAG baselines persists without in-distribution labeled retrieval, the paper could become a solid contribution; as it stands, the central claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mate, quick take on 2509.17455. The paper presents ICRAG, which wraps code-as-reasoning with retrieval and an execution/repair loop, and reports large gains over text baselines across 13 benchmarks. The genuinely new part is the scale and the 'computables' analysis: the AST profiles and complexity metrics of the generated code are a fresh lens on what benchmark language demands. The individual components are known, but the integration and the domain coverage (law, medicine) are new.\n\nThe paper is clearly written, with prompts in the appendix and per-task results. The authors also admit a few limitations, which counts in their favor.\n\nThe soft spot is real and load-bearing. R2, the retrieval pool of 'similar solved examples,' is built from 4/5 of each benchmark via k-fold, and each snippet is generated from the question plus its gold answer. The k-fold only stops the model from retrieving the exact test instance; it does not stop label leakage by analogy. A test case can retrieve code that prints 'violation' for a nearly identical fact pattern and infer the label. The paper never compares ICRAG against a version that retrieves only from R1 or from a genuinely held-out training split, so the central claim that ICRAG beats text-only reasoning is confounded. RAG_NL and IRCoT also use R2, so they share the leakage; that softens the comparison within the RAG family, but the headline against Direct/CoT/CoC is not controlled.\n\nOther issues are smaller. ProofNet relies on an automatic grader with inconsistent naming (GPT-3 vs GPT-o3) and no validation, and the appendix example suggests the generated code checks a single instance rather than proving a general statement. The correctness metric for nongeneralizable tasks is admitted to be the same as accuracy. Code and data are not released. And the arXiv metadata title/abstract doesn't match the full text—minor but sloppy.\n\nMy call: send it to serious peer review, but make the author address the R2 leakage with an R1-only ablation and a proper train/test retrieval split before I'd trust the accuracy numbers. Worth a reading-group discussion on transductive leakage alone.","headline":"A well-built empirical systems paper whose central accuracy claim is confounded by transductive label leakage from the in-distribution retrieval pool; the computables analysis is the real contribution.","tokens_in":36057,"tokens_out":3853,"would_cite":false,"duration_ms":35758,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turning benchmark language into runnable Python programs lifts average accuracy by 32.7 percent, the paper claims.","keywords":["text-to-code","code-as-reasoning","retrieval-augmented generation","program synthesis","codification","computables","iterative refinement","LLM reasoning"],"falsifier":"Remove the R2 pool entirely, or swap it for snippets generated from a different benchmark with answers scrubbed, and rerun ICRAG on CAIL and PubMedQA; if accuracy falls to the R1-only or CoC level, the k-fold retrieval pool is the source of the gains, while if accuracy holds, the repair loop is doing the work. A second check is to split R2 by nearest-neighbor similarity and test whether accuracy is systematically higher on questions whose retrieved snippets share answer-encoding templates.","tokens_in":35029,"feed_emoji":"🐍","tokens_out":4588,"duration_ms":40296,"temperature":0.7,"pith_summary":"The paper asks how much of benchmark language an LLM can genuinely understand when full formal semantics are out of reach. Its answer is ICRAG, a system that compiles each natural-language task into a Python program, retrieves relevant rules and code snippets, executes the program, and iteratively repairs it until it runs and returns an answer. Across 13 benchmarks in math, causal reasoning, law, and medicine, ICRAG reports average accuracy 32.7 percentage points above text-only baselines, with the largest gains on the hardest sets. A sympathetic reader would take the contribution to be evidence that operational computables, programs whose execution is the reasoning, offer a practical, inspectable middle ground between formal proof semantics and black-box text.","feed_headline":"Runnable code beats text-only reasoning on 13 benchmarks","feed_subtitle":"ICRAG retrieves rules and repairs programs until they execute, lifting average accuracy by 32.7 percent.","key_machinery":"The load-bearing mechanism is the retrieve-execute-repair loop over a computable: an executable Python program induced from the task text, whose runtime behavior is taken as operational evidence that the program understands the language. ICRAG starts from an initial code draft, retrieves from a knowledge pool R1 (domain documents) and a k-fold construction R2 (code snippets generated from other instances of the same benchmark, with the current instance held out), executes the draft, uses exceptions and trace failures as repair signals, and asks the LLM for a refined program and a new retrieval query; convergence is declared when no query is produced, typically within three iterations for 96% of test queries.","core_discovery":"The central claim is that weakening formal semantics down to executable behavior does not lose the ability to solve hard natural-language tasks: ICRAG's iterative retrieval-and-repair loop produces programs that outperform one-shot code execution and text-only chain-of-thought on every one of the 13 evaluated benchmarks, and outperform iterative text RAG on all of them as well. The program is not a side effect of the answer; it is the reasoning. Each iteration executes the current candidate, lets runtime failures expose missing conditions and exceptions, retrieves the minimal domain text or code fragment that fills the gap, and rewrites the program; the process stops when the model emits no further query. The paper presents this as showing that codification, previously limited to math-like questions, extends to causal, legal, and biomedical reasoning.","pith_inferences":["A stricter test would remove R2 entirely or use snippets from a disjoint corpus: the paper's own GitHub-only experiment suggests generic code alone yields smaller gains, implying that part of ICRAG's edge comes from in-domain analogies rather than external knowledge per se.","The k=5 construction means improvements on benchmarks like CAIL could partly reflect the model recognizing answer patterns from similar solved instances; a follow-up could vary k and measure accuracy on instances whose nearest R2 neighbors share the gold answer.","Treating execution failure as a semantic signal suggests a cheap consistency probe: mutate an input and check whether the revised program still encodes the same rule, which the paper's correctness metric only samples for 50 programs per dataset.","If computables become a standard artifact, benchmark quality itself could be reassessed by code-complexity statistics, for instance by flagging tasks whose generated programs are trivial conditionals as less informative."],"forward_implications":["If ICRAG's results hold, law and medicine benchmarks become treatable as programming tasks, and their answers come with runnable artifacts an auditor can read and test.","Execution feedback, not just retrieval, is what pushes accuracy above iterative text RAG: the paper's IRCoT baseline also retrieves iteratively but never compiles to code, and ICRAG beats it on every dataset.","The bounded iteration count, N≤3 for 96% of queries, means the transparency gain does not require unbounded compute per question.","For algorithmic tasks the generated programs generalize: re-running them on modified valid inputs preserved or exceeded original accuracy, so the code has learned the rule rather than only the instance.","Harder domains produce measurably more complex computables, including deeper stacks and higher cyclomatic complexity, giving a code-based lens for comparing tasks."],"supporting_citations":[{"why":"Chain of Code is the closest prior codification method and the one-pass baseline ICRAG must outperform.","marker":"[5]"},{"why":"Program of Thoughts supplies the idea of separating computation from narrative reasoning in generated programs.","marker":"[6]"},{"why":"Program-Aided Language models establish the code-as-reasoning setting that ICRAG extends.","marker":"[7]"},{"why":"Retrieval-augmented generation is the retrieval component ICRAG integrates into its repair loop.","marker":"[8]"},{"why":"IRCoT is the iterative retrieval baseline whose purely textual refinement ICRAG beats on every dataset.","marker":"[25]"},{"why":"GSM8K is the principal math word-problem benchmark used for evaluation.","marker":"[26]"},{"why":"Big-Bench-Hard supplies the large general-reasoning benchmark where ICRAG shows a 53.9 percent relative gain.","marker":"[30]"},{"why":"CAIL is the legal judgment benchmark where ICRAG reports its largest legal-domain improvement.","marker":"[32]"},{"why":"PubMedQA is the biomedical benchmark where ICRAG reports a 140.6 percent relative accuracy gain.","marker":"[35]"}],"fun_headline_variants":["ICRAG's iterated code repair beats text-only on 13 benchmarks","Executable reasoning outperforms text on all 13 benchmark tasks","Run, fail, repair: ICRAG turns benchmarks into code that wins","From text to runnable code: ICRAG tops 13 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The largest claimed gains rest on the assumption that code snippets drawn from the other four-fifths of the same benchmark, each containing its gold answer, do not let the model recover the held-out answer by analogy; if that assumption fails, the reported improvements could be answer leakage rather than retrieval-and-repair understanding.","fun_headline_variants_meta":{"raw":{"variants":["ICRAG's iterated code repair beats text-only on 13 benchmarks","Executable reasoning outperforms text on all 13 benchmark tasks","Run, fail, repair: ICRAG turns benchmarks into code that wins","From text to runnable code: ICRAG tops 13 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1332,"prompt_tokens":876,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":492,"tokens_out":456,"duration_ms":4321,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:47:31.013530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove the R2 pool entirely, or swap it for snippets generated from a different benchmark with answers scrubbed, and rerun ICRAG on CAIL and PubMedQA; if accuracy falls to the R1-only or CoC level, the k-fold retrieval pool is the source of the gains, while if accuracy holds, the repair loop is doing the work. A second check is to split R2 by nearest-neighbor similarity and test whether accuracy is systematically higher on questions whose retrieved snippets share answer-encoding templates.","supporting_citations":[{"cited_title":"In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F","cited_arxiv_id":null,"evidence_quote":"Chain of Code is the closest prior codification method and the one-pass baseline ICRAG must outperform."},{"cited_title":"In: Proceedings of the 40th International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Program-Aided Language models establish the code-as-reasoning setting that ICRAG extends."},{"cited_title":"In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H","cited_arxiv_id":null,"evidence_quote":"Retrieval-augmented generation is the retrieval component ICRAG integrates into its repair loop."}],"review_version":2}