{"id":"159b4b5e-75fa-44c7-9209-bc31a87770ca","arxiv_id":"2412.21199","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HumanEval Pro and MBPP Pro test whether LLMs can first solve a base function and then call it inside a more complex solution, and frontier models lose about 10 to 15 points on this task.","lead":"This paper introduces a new code benchmark task called self-invoking code generation, where a model must first solve a simple function and then use it to solve a harder related problem. Across 20+ LLMs, scores drop 10 to 15 percentage points on these benchmark versions, and instruction-tuned models show smaller gains than they do on standard benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation never verifies that generated code actually calls the base function, so the observed pass@1 drop may measure problem difficulty rather than self-invocation ability.","rationale":"I read the paper in good faith. The benchmark construction pipeline is careful about canonical solutions and test-case quality, and the iterative human verification in Table 1 is a genuine strength. However, the evaluation protocol has a clear gap: it scores only final test-case execution and never checks whether model outputs call the base function. The reader's weakest assumption identifies exactly this gap, and I agree it is the load-bearing concern. A secondary issue I noticed independently is that the instruction-tuning claim is overbroad: OpenCoder-8B-instruct improves more on HumanEval Pro (+20.1) than on HumanEval (+17.0), and several MBPP pairs show larger improvements on Pro than on MBPP, so the abstract's 'marginal improvements' statement is not uniformly supported by Table 2. That is a real but secondary issue; the construct-validity problem is the single most load-bearing one because it affects both headline observations. The concrete test I propose—rerunning the official harness with an AST-based call-graph check and restricted pass@1—would settle whether the benchmark measures self-invocation or merely harder problems. Because the reader already recommended a conditional verdict with a call-structure check, my read does not change the verdict.","tokens_in":23459,"tokens_out":6218,"duration_ms":59578,"concrete_test":"Using the released CodeEval-Pro evaluation harness, rerun the Table 2 evaluations for at least the six models with the largest claimed self-invocation drops (o1-mini, GPT-4o, Claude-3.5-sonnet, DeepseekCoder-33B-instruct, Qwen2.5-Coder-32B-instruct, and DeepseekCoder-6.7B-instruct). Before executing each generated file, parse it with Python's ast module; identify the self-invoking entry function from the top-level assert calls, and build the call graph of user-defined functions within that file. Then compute pass@1 both as reported and restricted to passing solutions whose entry function transitively calls at least one other function defined in the same file, and report the absolute difference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the 10%–15% pass@1 drop (e.g., o1-mini at 96.2% on HumanEval vs. 76.2% on HumanEval Pro) measures self-invoking code generation—rests on the assumption that the evaluation actually requires and detects calls to the model's own base function. The prompt in Appendix F.2 instructs that 'the solution of the second problem requires single or multiple calls to the first solution,' and the canonical solutions in Section 3.2 were manually checked to call the base function. However, the evaluation itself, described in Section 4 and the scoring setup in Appendix F.2, only executes the final test cases; it never inspects whether the generated code contains such a call. A model can inline the base logic, define a completely separate helper, or solve the harder problem directly and still pass all assertions. In that case the reported gap reflects the extra difficulty of the Pro problems, not the specific ability to invoke one's own previously generated solution. This is the load-bearing assumption: without a call-structure check, neither the headline performance drop nor the instruction-tuning observation is established as evidence about self-invocation. The benchmark may still be a useful difficulty benchmark, but its construct validity as a self-invocation benchmark is unproven.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a new code-generation task, self-invoking code generation, in which a model must solve a base problem and then solve a related, more complex problem by calling the function it wrote for the base problem. The authors generate three benchmarks, HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro, by prompting DeepSeek-V2.5 to create harder variants of existing problems, then iteratively executing and manually reviewing canonical solutions and test cases. They evaluate more than twenty proprietary and open-source LLMs, reporting a consistent 10-15 point absolute pass@1 drop on the Pro benchmarks relative to their base counterparts (e.g., o1-mini at 96.2% on HumanEval versus 76.2% on HumanEval Pro), and they present error-type analyses and a chain-of-thought study. The central interpretation is that the performance gap measures LLMs' ability to invoke their own generated code, and that instruction-tuned models improve less on this ability than on standard code generation.","tokens_in":23798,"tokens_out":6899,"duration_ms":68979,"significance":"If the benchmarks validly require self-invocation, the paper makes a useful contribution: it provides a reproducible recipe for upgrading saturated benchmarks, public datasets and model checkpoints, consistent tables across three benchmark families, and a concrete failure-mode taxonomy. The reported drop is large and coherent across models, and the base-versus-instruct comparison is thought-provoking. However, the paper's central construct-validity claim is not yet established because the evaluation never verifies that generated code actually calls the base solution; without such verification, the observed gaps may simply reflect the higher difficulty of the Pro problems. The benchmark could still be valuable as a difficulty-scaled extension of HumanEval and MBPP even if the self-invocation interpretation is weakened, but the title, abstract, and Section 5 conclusions currently overstate what the experiments establish.","major_comments":[{"comment":"The evaluation protocol only executes the final test cases and never inspects whether the model's second solution calls the first solution. A solution that inlines the base logic, defines a separate helper, or solves the harder problem directly passes the same assertions. The prompt instruction in Appendix F.2 ('requires single or multiple calls to the first solution') is not enforced by the harness. Consequently, the 10-15 point pass@1 drops in Table 2 and the confusion-matrix analysis in Section 5.2 may reflect overall problem difficulty rather than self-invocation ability. This is load-bearing for the paper's central claim, so I request a static or execution-based check that passing solutions actually invoke the base function, or a report of the fraction of passing solutions that contain such a call, with conclusions re-stated accordingly.","section":"Section 4 and Appendix F.2"},{"comment":"The claim that instruction-tuned models show 'marginal improvements' on self-invoking tasks rests on absolute point differences between HumanEval and HumanEval Pro (e.g., Qwen2.5-Coder-32B-instruct improves by 26.8 points on HumanEval but only 8.5 points on HumanEval Pro). Because the Pro problems are more difficult, absolute gains are not a clean measure of instruction-tuning efficacy, and the pattern is not uniform: Qwen2.5-Coder-1.5B-instruct loses 3.7 points on HumanEval Pro while gaining 26.8 points on HumanEval. The ratio defined in Eq. (1) is never used in the analysis. Please provide a difficulty-controlled or normalized comparison, and adjust the conclusion if the pattern is not robust.","section":"Section 5.1 and Eq. (1)"}],"minor_comments":[{"comment":"Specify exactly how pass@1 is computed for API models when temperature=0.2; the text says greedy decoding for open-source models and temperature=0.2 for API models, which leaves unclear the number of samples and whether the reported numbers are single-sample pass rates.","section":"Section 4"},{"comment":"The column header 'HumanEval (+)' is ambiguous; clarify which columns correspond to HumanEval, HumanEval+, MBPP, and MBPP+.","section":"Table 2"},{"comment":"Using the line count of canonical solutions as a proxy for problem complexity is not validated; a more direct measure, such as the number of required calls or test-case difficulty, would better support the complexity claim.","section":"Appendix E"},{"comment":"The Limitations section should acknowledge that the evaluation does not enforce the self-invocation constraint; currently it only mentions the Python-only restriction and the diversity of the generated problems.","section":"Limitations"},{"comment":"The choice of a 50%-70% solve-rate window for selecting BigCodeBench-Lite problems is not justified; please state the rationale and any sensitivity of the downstream results to this window.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is valid and is the main basis for the major-revision recommendation. The paper is otherwise solid: the benchmarks are public, the model checkpoints are listed, and the reported drop is consistent across tables. Adding a call-structure check or reframing the contribution as a difficulty-scaled benchmark would address the load-bearing issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a useful addition to the code-benchmark family, but the headline claim doesn't survive contact with the evaluation. The new task format—solve a base problem, then solve a harder one that should call the first solution—is genuinely new, and the authors document a consistent 10–15 point pass@1 drop on Pro versions of HumanEval, MBPP, and BigCodeBench-Lite, plus a smaller instruction-tuning gain. The tables are detailed and the model/API list is complete, so the measurements are broadly reproducible.\n\nThe soft spot is load-bearing. The evaluation only runs the final test cases. It never checks whether the generated code actually contains a call to the model's own base solution. The canonical solutions do call the base function, and the prompt instructs models to do so, but nothing in the harness enforces it. A model that inlines the logic or solves the harder problem directly passes the same tests. So the observed drop may simply reflect that the Pro problems are harder, not that models lack the specific ability to self-invoke. The error analysis—NameError being the second most common error—suggests some models do attempt to call a base function and get the name wrong, but that's anecdotal, not a measurement of the construct.\n\nA few smaller issues: DeepSeek-V2.5 generated the benchmark and is also evaluated, which creates minor circularity risk, though the core comparisons use external baselines. The abstract's 'marginal improvements' is over-stated; several instruct models gain around 10 points on Pro. The GitHub link has no commit hash or dataset checksum in the preprint, so the release isn't verifiable as-is.\n\nNone of this is fatal to the empirical phenomenon. A harder HumanEval/MBPP variant is a useful resource on its own. But as a 'self-invoking code generation' benchmark, the construct validity is unproven. I'd send this to a serious venue, but the referee should require either a call-structure check in the evaluation (e.g., static analysis of the generated code) or a rename that fits what's actually measured. I wouldn't cite it for the self-invocation claim until that's fixed.","headline":"Harder code benchmark, but the self-invocation claim isn't valid without a call-structure check.","tokens_in":24239,"tokens_out":2722,"would_cite":false,"duration_ms":26650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLMs perform 10–15 percentage points worse on code-generation tasks that require the model to call its own earlier solution, and that instruction tuning barely narrows that gap.","keywords":["self-invoking code generation","code generation benchmark","HumanEval Pro","MBPP Pro","pass@1 evaluation","instruction tuning","chain-of-thought prompting","LLM code reasoning"],"falsifier":"Modify the evaluation harness to log and inspect each generated program for a syntactic or semantic call to the base function before running hidden tests. If programs that contain no such call pass HumanEval Pro at a rate comparable to programs that do, the benchmark measures overall difficulty rather than self-invocation; if they fail, the phenomenon is real.","tokens_in":23238,"feed_emoji":"📉","tokens_out":5934,"duration_ms":53896,"temperature":0.7,"pith_summary":"This paper introduces self-invoking code generation as a distinct task: the model first solves a base programming problem and then writes a second, harder function that must call the first function to complete. To study this, the authors build HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro by generating more complex, semantically related variants of existing problems with verified canonical solutions and test cases. On these benchmarks, more than twenty models show a consistent 10–15 percentage-point pass@1 drop relative to the original HumanEval and MBPP, and the gap persists with one-shot prompting. The paper also finds that instruction-tuned models improve far less on self-invoking tasks than on standard ones, and that chain-of-thought prompting recovers only part of the loss. The result matters because it identifies a measurable capability—using one's own generated code as a building block—that current training and evaluation do not capture.","feed_headline":"LLMs lose 10–15 points when code must call their own solution","feed_subtitle":"New HumanEval Pro and MBPP Pro benchmarks show a capability gap that instruction tuning barely closes.","key_machinery":"The load-bearing object is the self-invoking code generation task, a paired-problem setup in which a model must solve a base problem and then a second, harder problem whose reference solution calls the first solution one or more times. The benchmark-construction pipeline generates the harder problems with a general-purpose code model, then iteratively executes candidate solutions with Python and manually reviews and fixes them until all canonical solutions pass the full test suite. Evaluation uses the standard pass@1 metric with the prompt instructing the model to write a Python file whose second solution requires single or multiple calls to the first, and error analysis classifies failures into assertion, name, value, index, type, and other runtime errors.","core_discovery":"The central claim is that current large language models are much better at generating a standalone function than at generating a related harder function that builds on it, and the paper calls this deficit \"self-invoking code generation.\" The benchmarks operationalize the task by pairing a base problem with a self-invoking problem and asking the model to write both solutions in one response, with the second solution explicitly expected to make single or multiple calls to the first. The paper reports that o1-mini passes 96.2% of HumanEval but only 76.2% of HumanEval Pro, and that across the model family the decline is concentrated in failures on the harder self-invoking problems rather than in the base problems. It further claims that instruction-based fine-tuning, which lifts base models by large margins on HumanEval, gives only marginal gains on HumanEval Pro and MBPP Pro, suggesting that existing training recipes do not target this reuse capability.","pith_inferences":["A testable extension the authors did not run: instrument the evaluator to require, in addition to passing hidden tests, that the emitted code actually contains a call to the base function; this would separate \"reuses own code\" from \"solves a harder related problem directly.\"","If self-invocation is the real bottleneck, then techniques that encourage decomposition—such as planning prompts, explicit function signatures, or training on compositional execution traces—should move HumanEval Pro scores more than further scaling of base-model data.","The observed NameError pattern suggests one concrete mechanism: models fail to carry the required function name or signature from the base problem into the second function, a determinable failure that could be fixed by consistency-aware decoding or prompt design.","Because the benchmarks are derived from a single generator model, part of the measured gap could be an artifact of how that generator phrases self-invoking problems; a multi-generator variant would show whether the 10–15 point drop is stable across problem phrasing."],"forward_implications":["Scores on HumanEval and MBPP overstate practical code generation ability, because a large share of the problems most models solve directly are not solved when the task requires composing a solution from the model's own code.","Instruction tuning as currently practiced is not sufficient for self-invoking code generation; closing this gap will require training data or objectives that specifically reward calling an earlier generated function.","Chain-of-thought prompting can be used as a partial mitigation: several models gain a few points on HumanEval Pro when asked to reason step by step, and the co-occurring drop in NameErrors suggests the reasoning makes generated code more self-contained.","The construction recipe is reproducible on other code benchmarks, as demonstrated by BigCodeBench-Lite Pro showing the same pattern, so the task can be extended as older benchmarks saturate."],"supporting_citations":[{"why":"Supplies HumanEval, the base benchmark that the new problems expand, and the pass@k metric used throughout.","marker":"Chen et al., 2021"},{"why":"Supplies MBPP, the second base benchmark expanded into MBPP Pro.","marker":"Austin et al., 2021"},{"why":"Supplies BigCodeBench, from which the authors derive BigCodeBench-Lite and BigCodeBench-Lite Pro for the generalization study.","marker":"Zhuo et al., 2024"},{"why":"Motivates the iterative test-case verification by showing that even strong models are unreliable at predicting test outputs.","marker":"Gu et al., 2024"},{"why":"Used to obtain rigorous HumanEval+ and MBPP+ comparison scores that frame the performance drop against stricter versions of the base benchmarks.","marker":"Liu et al., 2024"},{"why":"The generator model used to create self-invoking problems, candidate solutions, and test inputs in the construction pipeline.","marker":"DeepSeek-AI, 2024"},{"why":"Provides the chain-of-thought prompting method tested in the Section 5.3 experiments.","marker":"Wei et al., 2022"}],"fun_headline_variants":["Self-invoking code tasks stump LLMs, even with tuning","Code reuse: LLMs drop 20 points on chained problems","New benchmarks reveal weak link: calling your own code","LLMs ace simple code but fail when reusing solutions","Instruction tuning won't close the code reuse gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scoring only runs final hidden tests, so a model that solves the harder problem without ever calling its base-solution function still passes; the benchmark therefore measures whether the model can solve a harder related problem, not necessarily whether it actually reuses its own code.","fun_headline_variants_meta":{"raw":{"variants":["Self-invoking code tasks stump LLMs, even with tuning","Code reuse: LLMs drop 20 points on chained problems","New benchmarks reveal weak link: calling your own code","LLMs ace simple code but fail when reusing solutions","Instruction tuning won't close the code reuse gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1259,"prompt_tokens":999,"completion_tokens":260,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":178}},"tokens_in":615,"tokens_out":260,"duration_ms":2927,"temperature":1.0,"reasoning_tokens":178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:59:41.700095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Modify the evaluation harness to log and inspect each generated program for a syntactic or semantic call to the base function before running hidden tests. If programs that contain no such call pass HumanEval Pro at a rate comparable to programs that do, the benchmark measures overall difficulty rather than self-invocation; if they fail, the phenomenon is real.","supporting_citations":[],"review_version":1}