REVIEW 30 cited by
Unit Test Case Generation with Transformers and Focal Context
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Unit Test Case Generation with Transformers and Focal Context
read the original abstract
Automated unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult for developers to read or understand. In this paper we propose AthenaTest, an approach that aims to generate unit test cases by learning from real-world focal methods and developer-written testcases. We formulate unit test case generation as a sequence-to-sequence learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. Both techniques provide improvements in terms of validation loss, with pretraining yielding 25% relative improvement and focal context providing additional 11.1% improvement. We also introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 780K test cases mined from 91K open-source repositories from GitHub. We evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7% of the focal methods with only 30 attempts. We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3, finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated tests, showing overwhelmingly preference towards AthenaTest.
Forward citations
Cited by 30 Pith papers
-
Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
Sakura is a multi-agent system that generates structurally complex tests from NL descriptions, achieving 50-78% higher compilability and 38-66% higher coverage overlap than baselines on 1,464 scenarios from 20 Apache ...
-
From Exploration to Specification: LLM-Based Property Generation for Mobile App Testing
PropGen automates property generation for Android app testing via LLM synthesis from guided exploration and feedback refinement, yielding 912 valid properties and 25 previously unknown bugs across 12 apps.
-
An Iterative Test-and-Repair Framework for Competitive Code Generation
FixAudit improves LLM code generation on competitive programming benchmarks by training a shared model for iterative code-aware test generation and repair, achieving 35%+ gains in Pass@1 over baselines on the same 7B model.
-
CodeT: Code Generation with Generated Tests
CodeT improves code generation accuracy by using the same model to create test cases and then selecting solutions via output agreement on those tests, raising HumanEval pass@1 from 47% to 65.8%.
-
SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation
A contextual-bandit supervisor learns to route coding agents among default generation, analysis-augmented generation, and stop, yielding large coverage gains over unsupervised agents and non-agentic LLM test generators.
-
Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation
CATGen improves LLM unit-test reliability by combining structured project-context retrieval, deterministic test-class skeletons, and static-analysis repair, beating six baselines on compilation success, coverage, and cost.
-
Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows
A three-agent LLM workflow plus a test-specialized knowledge graph generates unit tests that beat prior LLM and search-based tools on coverage and mutation score.
-
Teaching Code LLMs to Reason with Intermediate Formal Specifications
Verification-guided fine-tuning with mutants and refinement traces teaches CodeLLMs to emit intermediate executable assertions that improve checking and repair on HumanExec.
-
TATG: Tracking-Aware Testing Objective for LLM-based Test Generation
Tracking individual static and feedback testing objectives across a coverage-then-mutation two-stage LLM workflow raises line/branch coverage and mutation score on complex Java methods versus KTester and PANTA.
-
Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
A multi-agent framework generates compilable, structurally complex Java tests from natural-language descriptions, beating Gemini CLI by 50–78% in compilability and 38–66% in coverage overlap.
-
FeedbackLLM: Metadata driven Multi-Agentic Language Agnostic Test Case Generator with Evolving prompt and Coverage Feedback
FeedbackLLM uses line and branch coverage feedback agents in an iterative multi-agent process with a redundancy cache to generate test cases achieving higher coverage than baselines on standard C and Python benchmarks...
-
Generalizing Test Cases for Comprehensive Test Scenario Coverage
TestGeneralizer generalizes an initial test into a set of executable tests covering more diverse scenarios, delivering +31.66% mutation-based and +23.08% LLM-assessed scenario coverage gains over ChatTester on 12 open...
-
Enhancing Program Repair with Specification Guidance and Intermediate Behavioral Signals
SpecTune improves LLM-based automated program repair by validating localized postconditions and using their intermediate violations for more precise fault localization and patching.
-
Enhancing Program Repair with Specification Guidance and Intermediate Behavioral Signals
SpecTune improves LLM-based automated program repair by deriving localized postconditions at execution checkpoints and using alpha and beta signals to produce precise fault-localization and patch-generation guidance.
-
An Iterative Test-and-Repair Framework for Competitive Code Generation
A four-stage RL framework that iteratively repairs one code candidate via a code-reading Auditor and a Fixer outperforms larger models and CURE/Specine on competitive programming Pass@1.
-
TestDecision: Sequential Test Suite Generation via Greedy Optimization and Reinforcement Learning
By proving test suite coverage is monotone submodular and training LLMs with RL to maximize marginal gains, TestDecision improves branch coverage 38-52% and bug detection up to 95% over base models on ULT and LiveCodeBench.
-
SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference
A feedback-driven multi-turn prompting framework outperforms single-pass LLM postcondition generation in correctness and completeness on EvalPlus and FixEval.
-
Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective
ARC compiles DSL-based requirement documents into runnable web apps by generating interface tests top-down, then using those tests to gate bottom-up code generation, achieving higher GUI pass rates than baseline codin...
-
SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
SAGE generates gray-box game regression tests via LLM-guided RL, compresses them with Pareto optimization, and prioritizes them via LLM-analyzed update logs, beating automated baselines on unique-bug detection at lowe...
-
Hamster: A Large-Scale Study and Characterization of Developer-Written Tests
A large-scale study of 1.7M developer-written Java tests shows they commonly use fixtures, mocking, structured inputs, and multi-class scope, features largely missing from automated test generation tools.
-
Generating Project-Specific Test Cases with Requirement Validation Intention
IntentionTest retrieves a reusable test from the project and edits it with an LLM to match a supplied validation intention, yielding tests that kill 28.1-37.6% more mutants, share 16.9-23.9% more coverage, and produce...
-
Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
PGS generates property-oriented, structurally minimal feedback from high-level program properties to refine LLM code, yielding up to 13.4% pass@1 gains and 1.4-1.6x higher bug-fix rates than prior TDD and debugging baselines.
-
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
CodeXGLUE supplies a standardized collection of 10 code-related tasks, 14 datasets, an evaluation platform, and BERT-, GPT-, and encoder-decoder-style baselines.
-
ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports
A tool-augmented multi-stage agent reproduces 58–70% of real GitHub issues as fail-to-pass tests, outperforming prior prompt and agent baselines at about $0.14 per issue.
-
Quantize with Confidence? An Empirical Study of Quantization for Code Generation
4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.
-
Leveraging Large Language Models to Obscure Code Stylometry: A Comparative Study of GPT-3.5 and GPT-4
GPT models with structured prompts can obscure code stylometry to varying degrees, but often at the cost of code functionality.
-
PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
PPO-LLM adaptively selects among eight prompting techniques using an 11-dimensional state vector to guide an LLM toward higher branch and line coverage than static baselines on 20 benchmark programs.
-
Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation
KTester improves LLM-generated Java unit tests on a 110-method benchmark by injecting static project knowledge and decoupling test-case design from test-method generation, beating four baselines.
-
StarCoder: may the source be with you!
StarCoderBase matches or beats OpenAI's code-cushman-001 on multi-language code benchmarks; the Python-fine-tuned StarCoder reaches 40% pass@1 on HumanEval while retaining other-language performance.
-
CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
CodePori is a multi-agent LLM system for code generation whose participant evaluation identifies practical challenges like memory limits and hallucinations missed by binary benchmarks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.