Pith. sign in

REVIEW 12 cited by

LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.10304 v2 pith:NVVTOKJD submitted 2024-04-16 cs.SE cs.LG

Kaibo Liu , Zhenpeng Chen , Yiyang Liu , Jie M. Zhang , Mark Harman , Yudong Han , Yun Ma , Yihong Dong
show 2 more authors
Ge Li Gang Huang
This is my paper · ORCID
classification cs.SEcs.LG
keywords bugstesttrickcatcherplausibleprogramsprogramdetectinginputs
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Detecting tricky bugs in plausible programs, those that pass existing test suites yet still contain bugs, remains a significant challenge in software testing. To address this problem, we propose TrickCatcher, an LLM-powered approach to generating test cases for uncovering bugs in plausible programs. TrickCatcher operates in three stages: First, it uses an LLM to generate program variants based on the program under test (PUT) and its specification. Second, it employs an LLM to construct an input generator from the specification for producing test inputs. Finally, these inputs are executed on both the PUT and its program variants to detect inconsistencies in their outputs. We evaluate TrickCatcher on two datasets, TrickyBugs and EvalPlus, which include 366 human-written and 151 AI-generated plausible programs with tricky bugs. TrickCatcher achieves recall, precision, and F1 scores that are 1.80x, 2.65x, and 1.66x those of the state-of-the-art baselines, respectively. Code and data used are available at https://github.com/RinCloud/TrickCatcher.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.

  2. COFFE: A Code Efficiency Benchmark for Code Generation

    cs.SE 2025-02 conditional novelty 7.0 of 10

    A new benchmark, COFFE, uses stressful test cases and CPU instruction counts to show LLM-generated code is often correct but time-inefficient.

  3. Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLMs, especially GPT-4, outperform existing fault localisation and repair tools for deep learning models in accuracy, speed, and stability.

  4. Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education

    cs.CY 2025-05 conditional novelty 6.0 of 10

    Across four LLMs, pitfall recall in machine-learning code averaged under 50%, with information-leakage and model-selection errors most often missed.

  5. Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding

    cs.SE 2025-04 conditional novelty 6.0 of 10

    Including the bodies of helper methods in the prompt enables large language models to generate better comments for dependent Java methods than standard comment generators.

  6. CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification

    cs.SE 2025-02 conditional novelty 6.0 of 10

    CLOVER is a new Python unit-test-generation benchmark with 845 problems, contexts up to 128k tokens, and coverage-based retrieval, where all 14 evaluated models score below 35% on the hardest task.

  7. Mutation-Guided LLM-based Test Generation at Meta

    cs.SE 2025-01 conditional novelty 6.0 of 10

    ACH generates concern-specific mutants and tests that kill them; its industrial deployment shows 73% engineer acceptance and 36% privacy relevance.

  8. Harnessing Large Language Models for Seed Generation in Greybox Fuzzing

    cs.CR 2024-11 conditional novelty 6.0 of 10

    SeedMind generates fuzzing seeds by having an LLM write a seed generator and iteratively refine it using code coverage feedback.

  9. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

  10. VeriLoC: Line-of-Code Level Prediction of Hardware Design Quality from Verilog Code

    cs.AR 2025-06 conditional novelty 5.0 of 10

    A method that predicts line-level timing and congestion issues directly from Verilog code using CL-Verilog embeddings and gradient-boosted classifiers.

  11. Black-Box Bug-Amplification for Multithreaded Software

    cs.SE 2025-07 reject novelty 4.0 of 10

    An ensemble classifier that predicts bug probability from input parameters substantially outperforms random sampling and other search heuristics for triggering rare concurrency bugs in a synthetic benchmark.

  12. PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization

    cs.SE 2025-05 reject novelty 4.0 of 10

    PRIMG combines mutant prioritization with LLM-based iterative test refinement, and its evaluation on three Solidity projects suggests prioritized mutant targeting yields more killed mutants than random targeting.

Pith tools