{"id":"149ff77f-fe08-4e43-b7a7-8436c5196470","arxiv_id":"2501.13978","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CGO, a concise goal-oriented prompting method, achieves accuracy comparable to or better than existing prompting baselines on code generation benchmarks while using fewer intermediate tokens.","lead":"This paper introduces Chain of Grounded Objectives (CGO), a two-stage LLM prompting method that first generates comment-style functional objectives from a problem description, then uses those objectives to guide code generation. It reports that CGO matches or beats several process-oriented prompting baselines on code benchmarks while producing fewer intermediate tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline fairness is the load-bearing risk: manually curated baseline prompts may understate existing methods, so the claimed accuracy advantage over CodeCoT and Self-Planning is unverified.","rationale":"The reader identified the zero-shot objective generation quality as the weakest assumption. That is a valid limitation, and the paper itself notes it, but it concerns the mechanism that makes CGO work, not whether the central comparative claim is true. The empirical comparison could be valid even if objectives are unmeasured. The more load-bearing risk is that the baselines are not faithful implementations of the published methods. Appendix A openly states that CoT and Self-Planning reasoning chains were manually curated for this paper's configuration, and the CodeCoT example in Table 10 contains a flawed reasoning chain that would produce incorrect code. If the baselines are weakened by construction, the reported accuracy gains of CGO (e.g., 4.9 points over CodeCoT on HumanEval with LLaMA3-8B) do not establish superiority over the actual methods. This directly attacks the central claim, whereas objective quality only affects the explanatory story. The proposed test—re-running with official baseline prompts—is concrete, low-cost, and would settle the concern. The reader's verdict of CONDITIONAL remains appropriate; the concern reinforces the need for additional verification before acceptance.","tokens_in":17299,"tokens_out":9513,"duration_ms":86326,"concrete_test":"Re-run the HumanEval and MBPP experiments for CGO and at least CodeCoT and Self-Planning on LLaMA3-8B-Instruct and LLaMA3-70B-Instruct, using the official prompt templates from the original papers (Huang et al. 2023 for CodeCoT; Jiang et al. 2024 for Self-Planning) with no manual curation. If CGO still outperforms these official baselines by at least the reported margins on the same benchmarks, the concern is resolved; if not, the comparative claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is comparative: CGO outperforms existing prompting techniques. This depends on the baselines being faithful implementations of those techniques. Appendix A states that for CoT and Self-Planning, 'we manually curated new examples tailored to our experimental configuration' rather than using the original methods' prompts. The CodeCoT example in Table 10 illustrates the risk: the reasoning chain instructs the model to 'split the input string by spaces' for the nested-parentheses problem, producing code that cannot correctly group adjacent parentheses—the shown output for '( ) (( )) (( )( ))' would be incorrect. If such underpowered prompts are representative, the reported CodeCoT scores (e.g., 57.5 pass@1 on HumanEval with LLaMA3-8B) are not a fair baseline. Self-Pseudo is also a custom three-stage method, not a standard published baseline, and no official prompts or code are released. If baselines were run with their original, published prompts, the margins between CGO and the baselines could shrink or reverse, directly undermining the 'outperforms' claim. The paper's own Limitation section does not acknowledge this risk, instead focusing on objective-generation quality, which is a secondary concern about mechanism rather than comparative validity.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chain of Grounded Objectives (CGO), a two-stage prompting method for LLM code generation: first, the model zero-shot generates a compact set of 'functional objectives' from the problem description; second, these objectives are prepended as comment-style context to the original problem for final code generation. The method is evaluated on HumanEval, MBPP-sanitized, HumanEval+/MBPP+, and LiveCodeBench with five LLMs (LLaMA3-8B/70B, LLaMA3.1-8B/70B, GPT-3.5-Turbo) against direct prompting, few-shot, CodeCoT, zero-shot CoT, Self-Planning, and Self-Pseudo, using pass@1 and pass-ratio@10. The main claims are that CGO matches or exceeds baseline accuracy while generating substantially fewer intermediate tokens.","tokens_in":17523,"tokens_out":6338,"duration_ms":52847,"significance":"If the comparative results are credible, CGO is a valuable, simple addition to the prompting toolbox for code generation: it demonstrates that concise, comment-style functional objectives can substitute for lengthy process-oriented reasoning, with a clear efficiency benefit. The paper ships all resources publicly, includes a broad evaluation across datasets and model scales, and tests the robustness of the wording of the objective-generation prompt. However, the significance of the accuracy claims is contingent on the faithfulness of the baseline implementations and on the statistical interpretation of the reported metrics.","major_comments":[{"comment":"The manual curation of baseline prompts is a load-bearing threat to the central comparative claim. Appendix A states that for CoT and Self-Planning, 'we manually curated new examples tailored to our experimental configuration' rather than using the original methods' published prompts. Table 10 illustrates the risk: the CodeCoT reasoning chain for the nested-parentheses problem instructs the model to 'Split the input string by spaces', which yields code that fails on inputs with nested groups such as '( ) (( )) (( )( ))' (the generated code returns a flat split list, not the required list of balanced groups). If similarly underpowered prompts were used across the benchmark, the reported margins over CodeCoT (e.g., 57.5 vs 62.4 pass@1 on HumanEval with LLaMA3-8B, Table 1) would not reflect the method's true performance. Please re-run the baselines with the original implementations/prompts from the cited papers, or provide evidence that the curated prompts are at least as strong as the originals.","section":"Appendix A / Table 10"},{"comment":"The metric labels 'pass@1 (n=10)' and 'pass-ratio@10' are misleading because the experiments use greedy sampling (temperature=0, top-p=1). With temperature=0, repeated runs produce identical outputs; the stated procedure of repeating the experiment 10 times per problem therefore yields 10 identical code solutions, so pass@1 (n=10) is simply the deterministic pass/fail of that single output, and pass-ratio@10 averages identical per-problem ratios. The n=10 multiplicity adds no statistical information. Please either sample with nonzero temperature for n=10 and compute the unbiased pass@k estimate, or report deterministic pass@1 and state explicitly that n=10 is not a sample size.","section":"Section 4.1.3 / Appendix B.1"},{"comment":"The reported differences are small and no measure of uncertainty is provided. For instance, on HumanEval with LLaMA3-70B (Table 2), CGO pass@1 is 80.3 versus 79.5 for Self-Planning and 79.3 for Zero-shot CoT; with 164 problems these ~1-point gaps are within the range of prompt-selection noise. Please provide confidence intervals (e.g., bootstrap) or a paired significance test across problems to support the claim that CGO 'outperforms' these baselines, rather than merely achieving a higher point estimate.","section":"Tables 1–3"}],"minor_comments":[{"comment":"There is a typo: 'bellow' should be 'below'.","section":"Section 4.1.3"},{"comment":"A pass@1 value of 40.75 is reported for GPT-3.5-turbo; since pass@1 counts solved problems, a fractional value should be accompanied by a definition (e.g., average over 10 samples). Clarify how this value is computed.","section":"Table 4"},{"comment":"The statement 'CGO consistently achieves high accuracy, outperforming baseline prompting techniques' is too strong; Table 4 shows Self-Planning outperforms CGO on LiveCodeBench with LLaMA3.1-8B (30.0 vs 28.0 pass@1). Qualify the claim to reflect the settings where CGO does not outperform.","section":"Conclusion"},{"comment":"The title uses 'Chain of Grounded Objectives' while the abstract introduces 'Concise Goal-oriented Prompting'; align the terminology for consistency.","section":"Abstract/Title"},{"comment":"ArchCode [10] is cited but not discussed; add a sentence of context in Related Work or remove the reference.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The paper is within scope for ECOOP as a prompt-engineering technique for code generation. The main risk is the baseline-fairness issue, which could be resolved with additional experiments or careful justification. The efficiency analysis and breadth of evaluation are strengths. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou asked about arXiv:2501.13978. The short version: CGO is a modest but reasonable idea—generate a small set of comment-style functional objectives from the problem, then feed those objectives plus the original description to the model for code generation. It's not a new paradigm; it's close in spirit to ArchCode and self-planning, but the comment-style format and the emphasis on token efficiency are the specific contributions. The paper does several things well: it tests across five benchmarks and five LLMs, reports pass@1 and pass-ratio@10, includes a difficulty breakdown, and consistently shows that CGO generates far fewer intermediate tokens than CoT/self-planning variants while holding accuracy. That efficiency result is the strongest part of the paper and is believable.\n\nThe soft spots are real, and one is load-bearing. First, baseline fairness: Appendix A says that for CodeCoT and Self-Planning they 'manually curated new examples tailored to our experimental configuration' rather than using the original prompts. Table 10's CodeCoT example is actually a bad prompt for the parentheses problem—it tells the model to split by spaces, which breaks groups with internal spaces. If such underpowered prompts are typical, the reported margins over CodeCoT (e.g., 57.5 vs 62.4 pass@1 on HumanEval with LLaMA3-8B) may be inflated. This is a direct threat to the 'outperforms' claim. Second, the metric reporting: 'pass@1 (n=10)' is confusing. Greedy sampling with temperature=0 gives deterministic outputs; generating ten objective samples per problem doesn't automatically give ten distinct code outputs unless there is actual stochasticity. The paper says they repeated the experiment 10 times due to floating-point/nondeterminism, which is not a principled way to get a distribution. No significance tests or error bars accompany the numbers, so the small margins (e.g., 80.3 vs 79.9) are hard to interpret. Third, no comparison with ArchCode, which is the most closely related requirement-based method; the paper cites it but doesn't test against it. The wording-variation experiment (objective vs requirement) is not a substitute. Also no code link is provided despite the claim of public availability.\n\nThe limitation section acknowledges that zero-shot objective generation may vary in quality, but it does not acknowledge the baseline-fairness risk, which I think is more important.\n\nBottom line: the efficiency advantage is solid; the accuracy advantage is conditional. This paper is worth a serious referee's time—the core idea is practical and the empirical scope is broad—but it needs a redo of the baseline comparisons with original prompts, clearer sampling/methodology, and ideally an ArchCode comparison. I would send it to review but with a request for major revision.","headline":"CGO is a simple, efficient goal-oriented prompting variant with plausible accuracy gains, but the evaluation's baseline fairness and metric reporting leave the comparative claims shaky.","tokens_in":18024,"tokens_out":3807,"would_cite":false,"duration_ms":38892,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain of Grounded Objectives (CGO) embeds comment-style functional objectives into the prompt, achieving code-generation accuracy on par with or better than process-oriented prompting while cutting intermediate tokens by more than half.","keywords":["code generation","prompt engineering","goal-oriented prompting","Chain of Grounded Objectives","chain-of-thought","functional objectives","token efficiency","pass@1"],"falsifier":"Take a set of coding problems, generate CGO objectives for each, have annotators (or a held-out reference model) rate each objective list for completeness and fidelity against the problem's test cases, and split problems into high-scoring and low-scoring objective groups. CGO's claim predicts that its pass@1 advantage over direct prompting is concentrated in the high-scoring group; if low-quality objectives still beat direct prompting (or if high-quality objectives do not), the supposed link between objective fidelity and code quality is refuted.","tokens_in":17111,"feed_emoji":"🎯","tokens_out":6913,"duration_ms":56244,"temperature":0.7,"pith_summary":"Chain of Grounded Objectives (CGO) is a two-stage prompting method for code generation. First, the LLM is asked to write a short set of comment-style functional objectives describing what the code must do. Then, the original problem plus that objective list is given to the model to generate the solution. The paper claims that this goal-oriented, 'what not how' format matches or beats process-oriented prompting (step-by-step reasoning, plans, pseudo-code) in pass@1 and pass-ratio@10 on five standard coding benchmarks, while cutting intermediate token counts by more than half. A reader should care because it suggests that verbose reasoning chains may be unnecessary for code generation; a concise, program-shaped statement of requirements is cheaper and often more accurate.","feed_headline":"Comment-style goals beat CoT for code, at half the tokens","feed_subtitle":"A two-stage prompt that writes functional objectives before code generation matches or exceeds all baselines on five public coding…","key_machinery":"The load-bearing mechanism is the objective-generation stage. The model is prompted with 'Write objectives to solve the problem,' and it returns a short list of natural-language clauses, each naming a functional requirement the solution must satisfy. Because the list is written in the register of code comments, it sits closer to the programming-language distribution the model saw in training, and it specifies 'what' without prescribing 'how.' This compact representation is then concatenated with the original problem when the code is generated. The two-stage composition O = GenObj(P), C = GenCode(O|P) carries the entire argument: if the objectives are faithful, the second prompt is a low-token distillation of the problem that behaves like an annotated function specification.","core_discovery":"On its own terms, the paper's discovery is that summarizing a coding problem into a compact set of functional objectives, formatted like the comments developers already write, gives the model better guidance than the procedural scaffolds that dominate current prompting practice. The method is formalized as C = GenCode(GenObj(P)|P), where GenObj turns the problem description P into objectives O and GenCode produces the program C from the enriched prompt. In experiments across several models of different sizes and five difficulty-scaled benchmarks, CGO generally achieved the highest or near-highest pass@1 and pass-ratio@10, and it did so while emitting far fewer intermediate tokens than the baselines. The paper interprets this as evidence that aligning the prompt with the structured, declarative conventions of programming languages is more effective than mimicking human step-by-step reasoning.","pith_inferences":["If objective fidelity is what drives the gain, then CGO should compose well with self-verification: after generating an objective list, the model could check its own code against each objective as a built-in test oracle, a direction the authors leave implicit.","The same comment-style goal-list trick may transfer to other structured generation tasks (data transformation, regex writing, SQL query synthesis) where the problem can be restated as a small set of functional clauses.","The paper's token-efficiency result suggests a prompt-selection rule of thumb: prefer the shortest intermediate representation that still names every hard constraint; process-oriented expansions should be reserved for problems where objectives alone are insufficient.","A direct testable extension: vary the number of objectives n and measure accuracy; if the method's benefit is tied to distillation, there should be an n at which accuracy peaks and then degrades as objectives become over- or under-specific."],"forward_implications":["CGO provides a low-cost alternative to Chain-of-Thought style prompts: equal or better accuracy with about half the intermediate tokens, which reduces inference time and cost.","Because the objective list is comment-shaped, it can be dropped into existing code-generation pipelines without execution or test-feedback loops, unlike test-driven prompting.","CGO's effectiveness holds across model sizes (mid-scale to large) and on harder, more realistic programming tasks, with the largest gains reported on the hardest difficulty tier.","The method is wording-robust: synonymous prompts ('objectives' vs 'requirements') shift results by about one percentage point, so it does not depend on exact phrasing."],"supporting_citations":[{"why":"Provides the primary function-completion benchmark used for the main evaluations.","marker":"[6]"},{"why":"Supplies the second benchmark and its problem format, which the paper standardizes to function-signature style.","marker":"[3]"},{"why":"Defines the extended test suites that add edge cases to the two main benchmarks, used for robustness evaluation.","marker":"[20]"},{"why":"Introduces the contamination-free, difficulty-tiered benchmark used for the real-world evaluation.","marker":"[13]"},{"why":"Defines the zero-shot Chain-of-Thought baseline that CGO is compared against and that often outperforms scripted CoT.","marker":"[17]"},{"why":"Introduces the self-planning baseline, the strongest process-oriented competitor in the paper.","marker":"[16]"},{"why":"Introduces the one-shot Chain-of-Thought baseline for code generation that the paper contrasts with CGO.","marker":"[11]"},{"why":"Defines the pass-ratio@n metric used to measure solution quality beyond pass@1.","marker":"[31]"}],"fun_headline_variants":["Objectives over steps: better LLM code, fewer tokens","Comment-style goals boost code generation","Skip CoT: functional prompts win for code","CGO beats chain-of-thought at half the tokens","Code prompts: goals, not procedures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's benefit depends on the zero-shot objective-generation step producing objectives that actually capture the problem's functional requirements without loss or distortion; if that step yields vague, redundant, or wrong objectives, the code-generation stage has no advantage over direct prompting and the efficiency gain is bought at the cost of accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Objectives over steps: better LLM code, fewer tokens","Comment-style goals boost code generation","Skip CoT: functional prompts win for code","CGO beats chain-of-thought at half the tokens","Code prompts: goals, not procedures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1362,"prompt_tokens":838,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":454,"tokens_out":524,"duration_ms":5728,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:15:09.881873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of coding problems, generate CGO objectives for each, have annotators (or a held-out reference model) rate each objective list for completeness and fidelity against the problem's test cases, and split problems into high-scoring and low-scoring objective groups. CGO's claim predicts that its pass@1 advantage over direct prompting is concentrated in the high-scoring group; if low-quality objectives still beat direct prompting (or if high-quality objectives do not), the supposed link between objective fidelity and code quality is refuted.","supporting_citations":[],"review_version":1}