REVIEW 18 cited by
Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recently, the ChatGPT LLM has received great attention: it can be used as a bot for discussing source code, prompting it to suggest changes, provide descriptions or even generate code. Typical demonstrations generally focus on existing benchmarks, which may have been used in model training (i.e., data leakage). To assess the feasibility of using an LLM as a useful assistant bot for programmers, we must assess its realistic capabilities on unseen problems as well as its capabilities on various tasks. In this paper, we present an empirical study of ChatGPT's potential as a fully automated programming assistant, focusing on the tasks of code generation, program repair, and code summariziation. The study investigates ChatGPT's performance on common programming problems and compares it with state-of-the-art approaches on two benchmarks. Among several findings, our study shows that ChatGPT is effective in dealing with common programming problems. However, our experiments also reveal limitations in terms of its attention span: detailed descriptions will constrain the focus of ChatGPT and prevent it from leveraging its vast knowledge to solve the actual problem. Surprisingly, we have identified the ability of ChatGPT to reason the original intention of the code. We expect future work to build on this insight for dealing with the open question of the oracle problem. Our findings contribute interesting insights to the development of LLMs for programming assistance, notably by demonstrating the importance of prompt engineering, and providing a better understanding of ChatGPT's practical applications for software engineering.
Forward citations
Cited by 18 Pith papers
-
Can LLMs Test Terminal User Interfaces?
Automated TUI testing is feasible but unsolved: random input wins on raw crash counts under a time budget, LLM guidance wins per interaction and on input-gated faults, and line coverage does not predict crash finding.
-
AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints
A multimodal, contrastive-learning system attributes GitHub PRs to five AI coding agents and humans with weighted F1 0.96, and shows that text style, not code, carries the attribution signal.
-
AdaptAgent: A Multi-agent, Domain-Guided Reasoning Framework for Code Adaptation
A multi-agent LLM pipeline that plans code adaptations using summarized intent, domain checklists, and sibling-method context outperforms single-shot prompting and repair baselines on Java adaptation examples.
-
AI-Assisted Modeling: DSL-Driven AI Interactions
A conversational, voice-enabled modeling workflow for the Lingua Franca DSL with instant diagram feedback and grammar-steered LLM tool calls.
-
ReCatcher: Towards LLMs Regression Testing for Code Generation
ReCatcher systematically measures regressions in LLM code generation across correctness, static quality, and performance, and its evaluation shows fine-tuning, merging, and new releases each introduce specific regressions.
-
On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
Even the best tested LLM judge, GPT-4-turbo, frequently misjudges code correctness (e.g., 50% of wrong Java functions judged correct) but agrees moderately with humans when judging code summary quality.
-
Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
Execution traces in prompts give inconsistent gains for LLM-based program repair, while LLM-optimized trace summaries improve consistency.
-
Inducing Vulnerable Code Generation in LLM Coding Assistants
A short string hidden inside a code comment can make LLM coding assistants generate attacker-chosen vulnerable code when they retrieve that comment from the web.
-
MoGraphGPT: Creating Interactive Scenes Using Modular LLM and Graphical Control
An interactive no-code system combining element-level modular LLM sessions, drawing-based graphical proxies, and automatic sliders creates 2D scenes faster than Cursor Composer in a 10-participant study.
-
CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
CallNavi is a new benchmark for LLM function calling with unfiltered, nested, multi-step API tasks; GPT-4o leads the leaderboard and a two-step routing pipeline improves fine-tuned models.
-
When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning
LLM-based sketch generation for embedded ML is fragile, with success rates below 40%, and prompt structure alone can swing outcomes from 15% to 30%.
-
Lossless Token Sequence Compression via Meta-Tokens
A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.
-
Assessing Large Language Models in Comprehending and Verifying Concurrent Programs across Memory Models
Five LLMs, including GPT-4, handle many simple concurrency checks but cannot reliably verify small programs under TSO and PSO relaxed memory models.
-
Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering
Interactive prompting (context enrichment, task decomposition, and question-asking) lifts LLM code snippet adaptation pass@1 from about 52% to 68-75% on ClassEval, with a multi-agent variant nearly matching human feedback.
-
A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering
PRIMES is a proposed four-stage checklist for prompt creation, pilot testing, multi-LLM evaluation, and output validation in LLM-based repository mining.
-
CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback
CHAI trains a reward model on GPT-4o preference labels and uses PPO to align Llama-3.1-8B for English-to-Hinglish translation, claiming a 25.66% human win-rate improvement over baselines.
-
The Impact of AI-Generated Solutions on Software Architecture and Productivity: Results from a Survey Study
In a 40-person practitioner survey, AI tools were reported to help most with small code snippets and to degrade architecture quality when applied to large, complex problems.
-
Resource-Efficient & Effective Code Summarization
QLoRA fine-tuning outperforms full fine-tuning for code summarization on CodeXGLUE while cutting GPU memory by roughly two-thirds.
Discussion (0). Continue with ORCID to comment.