REVIEW 8 cited by
Interactive Code Generation via Test-Driven User-Intent Formalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, when interacting with LLMs, users have no guarantees that the code suggestions produced correctly satisfy the intent they provided. In fact, it is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. In this paper, we propose the workflow of {\it interactive test-driven code generation}, which leverages lightweight user feedback to (a) formalize the user intent using generated tests that can be useful for debugging, and (b) produce an improved set of code suggestions by pruning and ranking candidate code suggestions. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder. We perform an automated evaluation of TiCoder on the \emph{MBPP} and \emph{HumanEval} code generation benchmarks. Our results are promising with using the OpenAI Codex LLM: our best algorithm improves the \passk{1} code generation accuracy (in absolute percentages) between $22.49\%$ to $37.71\%$ for MBPP and between $24.79\%$ to $53.98\%$ for HumanEval using between 1 to 5 simulated user queries.
Forward citations
Cited by 8 Pith papers
-
AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation
AssumptionMiner extracts implicit assumptions from LLM-generated code with a cross-model ensemble F1 of 0.816 on a new 180-task benchmark, and enables assumption-guided targeted regeneration.
-
Evaluating LLM-Based Regression Test Generation
A feedback-driven LLM tool generates regression tests for commits in minutes and, when its outputs seed a fuzzer, finds more bugs than a 24-hour state-of-the-art fuzzer on the same benchmark.
-
Specification Grounding Drives Test Effectiveness for LLM Code
Specification grounding, not test quantity, is the main reason AI-written tests improve LLM code: +38 pp correctness and 0% false alarms over a strong edge-prompted baseline.
-
Teaching Code LLMs to Reason with Intermediate Formal Specifications
Verification-guided fine-tuning with mutants and refinement traces teaches CodeLLMs to emit intermediate executable assertions that improve checking and repair on HumanExec.
-
SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference
A feedback-driven multi-turn prompting framework outperforms single-pass LLM postcondition generation in correctness and completeness on EvalPlus and FixEval.
-
CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback
CodeEvo uses two interacting LLM agents with keyword-guided instruction evolution and hybrid compiler-plus-LLM feedback to synthesize high-quality instruction-code pairs for fine-tuning code models.
-
DisFaceRep: Representation Disentanglement for Co-occurring Facial Components in Weakly Supervised Face Parsing
DisFaceRep reports a weakly supervised face parsing framework that disentangles co-occurring facial components and outperforms existing weakly supervised segmentation methods on CelebAMask-HQ, LaPa, and Helen.
-
Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents
Running multiple multi-agent teams in parallel, stopping at the first successful answer, cuts end-to-end latency by up to 2.2x with roughly unchanged completion rates on GAIA.
Discussion (0). Continue with ORCID to comment.