Pith. sign in

REVIEW 6 cited by

What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.08105 v3 pith:UVKZVCN3 submitted 2024-10-10 cs.CL

Kunhao Zheng , Juliette Decugis , Jonas Gehring , Taco Cohen , Benjamin Negrevergne , Gabriel Synnaeve This is my paper · ORCID
classification cs.CL
keywords modelscodegenerationlargelanguagemulti-turnmultipleperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A new spoken math benchmark, Spoken-MQA, shows that current speech-based AI models reason poorly from spoken math input, especially for arithmetic and knowledge-heavy problems.

  2. Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A minimal-prior pipeline with automated data curation and verifier-driven RL lets small LLMs generate verifiable Dafny specifications and beat larger proprietary models on a synthetic compositional benchmark.

  3. RedCoder: Automated Multi-Turn Red Teaming for Code LLMs

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A multi-turn red-teaming agent trained on simulated attacker-defender conversations induces vulnerable code at higher rates than prior attack methods across several code LLMs.

  4. Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.

  5. PBE Meets LLM: When Few Examples Aren't Few-Shot Enough

    cs.DB 2025-07 conditional novelty 5.0 of 10

    GPT-4o with multi-turn verification plus a Foofah fallback reaches 86.3% weighted accuracy on tabular PBE benchmarks, beating Foofah (57.1%) and Prose (47.3%).

  6. Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

    cs.CL 2025-05 conditional novelty 5.0 of 10

    EXSEARCH trains LLMs for agentic search by treating search trajectories as latent variables and optimizing a weighted likelihood via expectation-maximization, yielding gains on NQ, HotpotQA, MuSiQue, and 2WikiQA.

Pith tools