Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AUGER couples defect detection with attention-guided test generation to trigger 84 errors on Defects4J, outperforming five baselines without retraining the LLM.

desk verdict A concrete and potentially useful attention-steering recipe for LLM test generation, with a real but addressable evaluation gap: the attention head set is selected on a slice of the same benchmark used for the headline numbers. read the letter →

arxiv 2412.00828 v1 pith:T4KJPJ66 submitted 2024-12-01 cs.SE

Xin Yin , Chao Ni , Xiaodan Xu , Xiaohu Yang This is my paper · ORCID
classification cs.SE
keywords softwaredefectpredictionautomaticunittestgenerationerror-triggeringtestsattentionmechanismlargelanguagemodelspromptengineeringDefects4Jadversariallearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Software defect detectors usually output a binary flag, and automatic test generators generate many tests that never fail. This paper proposes AUGER, a two-stage method that treats the two problems as one: first predict which methods are defective and which statements within them are suspicious, then use that location information to steer a large language model's attention while it writes unit tests, so the tests target the predicted faulty lines. The central claim is that this self-guidance materially raises error-triggering efficiency: on Defects4J, AUGER triggers 84 method-level errors compared with 61 for the strongest baseline, and it also improves defect-detection F1-score, precision, and PR-AUC by double-digit percentages over four baselines. The 'What You See Is What You Get' name captures the design principle — the defect the detector sees is the defect the test generator is pushed to expose.

What carries the argument

The load-bearing mechanism is the attention projection in Equation 3: $\boldsymbol{H}^{(l,h)} = W(\boldsymbol{A}^{(l,h)}) \boldsymbol{V}$ with $[W(\boldsymbol{A})]_t = \boldsymbol{A}_t / C$ if token $t$ is in the defective statement set $S$ and $\alpha \boldsymbol{A}_t / C$ otherwise, where $C = \sum_{t \in S} \boldsymbol{A}_t + \sum_{t \notin S} \alpha \boldsymbol{A}_t$ renormalizes the scores. AUGER first profiles an LLM (DeepSeek Coder 6.7B or CodeLlama 7B) by modifying each attention head in turn and keeping the top $k=10$ heads that trigger the most errors on a 100-sample profiling set from Defects4J; at inference it applies the projection to those heads only. This changes where the model looks without changing any weights, so the guidance costs roughly the same as inference.

What would settle it

Concretely, select the attention head set H using a profiling set drawn from a different benchmark (e.g., Bears or Bugs.jar), then measure triggered errors on Defects4J's 723 methods; if the count falls to the no-guidance level of 67 errors, the attention profiling is overfitting the profile slice rather than transferring.

Watch

Extended reading notes

Core claim

On the paper's own terms, AUGER's discovery is that localizing defects and generating error-triggering tests can be welded into a single loop with no LLM fine-tuning. The detector is a UniXcoder encoder trained with adversarial and contrastive losses; it outputs both a method-level defect score and statement-level defect locations. The generator then applies an attention projection during inference that scales down the attention scores of tokens outside the predicted defective statements, renormalizes, and thereby makes the LLM focus on the suspicious lines. Evaluated on three Java bug benchmarks, the paper reports that AUGER improves F1-score by 4.7% to 35.3% and precision by 17.7% to 40.4% over LineVul, SVulD, CodeBERT, and UniXcoder for defect detection, and triggers 35 errors on detected Defects4J defects and 84 errors on all 723 Defects4J method-level defects, beating TOGA, EvoSuite, Randoop, AthenaTest, and CodeT5+ by 23 to 84 errors.

Load-bearing premise

The reported gains rest on the assumption that the 100 Defects4J samples used to choose which attention heads to modify are representative enough that the chosen heads also work on the rest of Defects4J and on unseen projects, rather than being overfit to that slice.

Editorial extensions

If this is right

  • Error-triggering test generation can be improved by a purely inference-time steering mechanism, so an LLM need not be fine-tuned or retrained to target known defects.
  • Defect location information does double duty: it explains the detector's output and narrows the search space for test generation, closing the loop between prediction and validation.
  • Using ground-truth defect locations (AUGER-gt) triggers 99 errors versus 84 with predicted locations, implying that better localization would yield still higher test-generation gains.
  • On real-world defect-fixing commits collected after the LLM's training cutoff, AUGER triggers 6 of 41 errors, suggesting the approach transfers beyond benchmark defects.
  • The combined adversarial and contrastive learning objectives are individually and jointly effective, with the full model reaching F1 0.276 and FPR 0.124 on Defects4J.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the attention head set is chosen on a 100-sample Defects4J slice, a stricter check of the approach would select heads on one benchmark (say, Bears/Bugs.jar) and report triggered errors on Defects4J; if the margin collapses, part of the reported gain is in-distribution selection.
  • The attention projection is model-agnostic and task-agnostic: any transformer-based code model could be steered toward known regions of interest, which suggests applications beyond testing, such as guided bug repair or focused code summarization, whenever a prior model identifies relevant tokens.
  • The reported precision of 8.8% on all Defects4J defects means about one in eleven generated tests triggers an error; combining attention guidance with coverage or mutation filtering might cut the number of tests developers must run.
  • AUGER-gt's 99 triggered errors versus AUGER's 84 gives a concrete ceiling: investing in better statement-level localization, e.g., by using the generator's own failing tests as supervised signal, is the most direct path to further gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes AUGER, a two-stage approach for method-level defect detection and error-triggering unit test generation. In the first stage, a UniXcoder encoder is fine-tuned with adversarial and contrastive learning to predict defective methods and to locate defective statements. In the second stage, AUGER profiles the attention heads of a large language model (DeepSeek Coder or CodeLlama) on a small subset of Defects4J, selects the top-k heads, and during inference re-weights attention scores toward predicted defective statements. The generated unit tests are automatically injected and executed on defective and fixed versions. The authors report improvements over four defect-detection baselines on Bears, Bugs.jar, and Defects4J, and report 35 and 84 triggered errors on the detected and all Defects4J method-level defects, respectively, outperforming five unit-test-generation baselines. A post-March-2023 real-world dataset is used to assess generalization, and an ablation study isolates the contribution of the attention-modifying mechanism.

Significance. If the results hold, AUGER provides a simple, training-free way to steer LLM generation toward defect-relevant code, potentially improving the efficiency of error-triggering unit test generation while also supplying developers with an explanation (the located defective statement). The paper has concrete strengths: it includes a public replication package, an out-of-distribution dataset collected after the LLM's knowledge cutoff to mitigate data leakage, and ablation studies for both the detection components (Table X) and the attention mechanism (Table XI). However, the central error-triggering claim is undermined by a selection-on-evaluation concern in the attention profiling step, and several methodological details are under-specified. The core idea is worth pursuing, but the current evidence does not yet establish that the attention guidance generalizes beyond the benchmark used to select the attention heads.

major comments (3)
  1. [Section III-B3 and Section V-B, Tables VII and XI] The attention head set H is chosen by evaluating each of the 1024 candidate heads on a 100-sample profiling set D sub-sampled from Defects4J, and the headline result of 84 triggered errors is then measured on the full Defects4J set of 723 method-level defects. Because D is likely included in the evaluation set, and because the paper does not report the profiling-set performance or an independent head-selection experiment (e.g., heads selected on Bears or Bugs.jar and transferred to Defects4J), the reported advantage of AUGER over AUGER_wo (84 vs. 67, Table XI) may be inflated by chance selection of heads that happen to trigger errors on D. This is a load-bearing issue: the 17-error difference is the main evidence that attention-based self-guidance is effective. The authors should re-run the evaluation with heads selected on a dataset disjoint from the evaluation set, exclude D from the test set, and report the variability of the triggered-error counts across multiple profiling samples.
  2. [Section III-A3, Eq. (2)] The contrastive learning component is described as using KL-divergence loss from R-Drop to minimize the distance between normal and adversarial samples, but R-Drop's loss is defined between two output distributions of the same input under different dropout masks; the paper does not specify the exact KL term, the probability distributions to which it is applied, or how FGM perturbations are constructed for code tokens. Since the ablation in Table X attributes a substantial part of the F1-score improvement to this component (0.242 to 0.276), the formulation must be precise enough for reproduction. Without this detail, the defect detection contribution is not fully verifiable.
  3. [Section III-B3 and Section V-B, Tables VII and IX] The paper claims that the attention head set H, once determined, can be used for both existing and unseen datasets, but the generalization evidence is incomplete. The real-world dataset evaluation (Table IX) reports only AUGER's recall and does not include the no-attention baseline AUGER_wo, so it is unknown whether the attention guidance transfers to out-of-distribution projects. Additionally, AUGER* (CodeLlama) is evaluated with the attention modification without stating whether the H selected on DeepSeek Coder is model-specific or transferred across architectures. The authors should either provide an ablation on the real-world dataset or explicitly discuss why the profiling on one model/dataset is expected to transfer.
minor comments (6)
  1. [Abstract and Section I] The improvement ranges differ between the abstract ('4.7% to 35.3% and 17.7% to 40.4%') and the introduction ('11.3% to 35.3%, 20.0% to 40.4%') for F1-score and Precision; please clarify which comparison (within-dataset or cross-dataset) is being reported.
  2. [Section IV-A, Table II] Table II reports 1,130 defective methods in Defects4J, while Section V-B evaluates on 723 method-level defects; the filtering process that yields 723 should be stated explicitly (e.g., methods with executable test infrastructure).
  3. [Section V-B, Table VIII] The column header 'Prop.' is not defined; it appears to denote the fraction of triggered errors, but the numerator and denominator should be described in the caption or text.
  4. [Section III-B3, Algorithm 1] The statement that attention profiling 'demands similar computational resources as inference' is misleading: profiling requires evaluating 1024 attention heads on a 100-sample set, which is orders of magnitude more expensive than a single inference pass and should be accounted for in the efficiency comparison of Figure 7.
  5. [Section V-B, Fig. 6] The text of Figure 6 contains rendering artifacts (repeated '/uni' strings) and the axis labels are unclear; the figure should be regenerated so that the candidate-number curves are legible.
  6. [Section V-C, Table XI] The description 'AUGER_wo denotes the elimination of the attention-modifying component' suggests ablation of the full attention mechanism, but the paper does not specify whether AUGER_wo still uses the same prompt with defective-method marking; this should be clarified so that the 17-error difference is attributable to attention reweighting rather than prompt engineering.

Circularity Check

2 steps flagged · score 6.0 of 10

Attention-head and candidate-count configurations are selected on a Defects4J profiling subset and then evaluated on the same Defects4J benchmark, so the headline error-triggering gain is partially a fitted result.

  1. fitted input called prediction [Section III-B3 (Task 3: Attention Profiling) and Section V-B (RQ-2, Table VII)]
    "Specifically, we sub-sample profiling set D (100 samples) from the Defects4J dataset (cf. Section IV-A)... We rank all the heads based on their unit test generation performance, specifically, by evaluating how many errors can be triggered on D. Subsequently, we define the attention head set H for inference as the top k performing heads... for the 723 method-level defects in Defects4J, TOGA, EvoSuite, Randoop_reg, Randoop_rev, AUGER*, and AUGER could trigger 61, 26, 39, 35, 78, and 84 errors, respectively."

    Algorithm 1 selects the attention head set H by measuring each candidate head's error-triggering count on D, and D is explicitly 'sub-sampled from the Defects4J dataset'. The headline error-triggering evaluation (Table VII) is then reported on 'the 723 method-level defects in Defects4J', with no statement that the 100 profiling samples are excluded. The 84 triggered errors therefore include the same distribution used to fit H; the ablation gain of AUGER over AUGER_wo (84 vs 67) is the evidence for the paper's central attention-guidance claim, but it compares a head set fitted to D against a no-attention baseline on an evaluation set overlapping D. This is selection-on-evaluation rather than an independent prediction of the value of the selected heads.

  2. fitted input called prediction [Section V-C (RQ-3, unit test candidate number) and Section IV-D (Experimental Setting)]
    "Fig. 6: The varying performance of AUGER and TOGA with different unit test candidate number on all defects... Different candidate numbers have varying impacts on AUGER's performance and the performance of AUGER increases as the number of candidates increases... Considering both the performance improvement and the generation cost caused by LLM, we adopt 100 unit test candidate numbers as the default setting."

    Figure 6 plots 'the varying performance of AUGER and TOGA with different unit test candidate number on all defects' — i.e., on the same 723 Defects4J defects that yield the headline 84-error result in Table VII. The paper then chooses the default candidate number (100) from this curve ('we adopt 100 unit test candidate numbers as the default setting'). Setting the candidate count by inspecting the evaluation benchmark, and then reporting the benchmark result at the chosen count, is a second configurable input fitted to the test distribution; it does not by itself invalidate the comparison with baselines (which use the same count), but it means the absolute triggered-error figures are not configuration-independent predictions.

full rationale

Most of AUGER's derivation is self-contained and non-circular. The defect-detection result on Defects4J is a genuine cross-dataset test: the model is trained and validated on Bears and Bugs.jar and only then tested on Defects4J (Section V-A), so that claim has independent content. The error-triggering pipeline also has independent components: prompts, statement-level defect location, unit-test validation, and the March-2023 real-world dataset are out-of-distribution checks. The circularity is concentrated in the configuration of the attention guidance. The paper selects the attention head set H by ranking heads on a 100-sample profiling set D drawn from Defects4J (Section III-B3, Algorithm 1), and then reports the headline 84 triggered errors on 'the 723 method-level defects in Defects4J' (Section V-B) without excluding D. The ablation showing the benefit of attention guidance (AUGER 84 vs AUGER_wo 67, Table XI) is therefore measured on the same benchmark used to pick the heads. The candidate count (100) is likewise selected after viewing the performance curve on 'all defects' (Section V-C). The out-of-distribution real-world study (Table IX) would be the natural way to show the selected H generalizes, but it omits the AUGER_wo baseline, so it cannot rescue the claim. These are concrete selection-on-evaluation steps; they do not make the whole method tautological, but they inflate — and could even account for — the central attention-guidance improvement. No load-bearing self-citation chain or imported uniqueness theorem is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The attention head set H is a subset of existing heads in the LLM and is treated as a configuration, not an invented entity. The free parameters are the attention scaling coefficient, the number of heads, the profiling set size, and the candidate count, all selected using the target benchmark's distribution.

free parameters (4)
  • alpha = 0.01
    Attention scaling coefficient in Eq. 3; set by hand and evaluated on the profiling set D from Defects4J (Section III-B3, Section IV-D).
  • k = 10
    Number of top attention heads kept in H; chosen during profiling on Defects4J subset (Section III-B3).
  • profile set size = 100
    Number of samples in the profiling set D sub-sampled from Defects4J (Section III-B3).
  • candidate number = 100
    Default number of candidate unit tests generated per method, chosen after evaluating 10 to 120 candidates on Defects4J (Section V-C).
assumptions (4)
  • domain assumption Fine-tuned UniXcoder embeddings reliably rank defective methods and locate defective statements
    The whole pipeline depends on the quality of transfer from UniXcoder pretraining to method-level and statement-level defect prediction (Section III-A, Section III-B).
  • domain assumption DeepSeek Coder and CodeLlama can generate executable Java unit tests from the prompt template
    The error-triggering stage assumes the LLM produces syntactically valid JUnit tests that compile after dependency injection (Section III-B2, Section III-C).
  • ad hoc to paper Reweighting attention scores according to Eq. 3 shapes generation toward defective statements without breaking fluency
    This is the core mechanism of AUGER; it is an empirical design choice validated only by the observed error-triggering counts, not by a formal derivation (Section III-B3).
  • domain assumption A test that fails on the defective version and passes on the fixed version is a valid error trigger
    The metric definition in Section IV-C and prior work treat the minimal patch as isolating the error; this assumes the test failure is due to the target defect and not test flakiness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation." pith.science (2026). https://pith.science/paper/T4KJPJ66

@misc{pith2026241200828,
  author       = {Pith},
  title        = {Pith review of: What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T4KJPJ66}},
  note         = {Machine review of arXiv:2412.00828}
}
read the original abstract

Software defects heavily affect software's functionalities and may cause huge losses. Recently, many AI-based approaches have been proposed to detect defects, which can be divided into two categories: software defect prediction and automatic unit test generation. While these approaches have made great progress in software defect detection, they still have several limitations in practical application, including the low confidence of prediction models and the inefficiency of unit testing models. To address these limitations, we propose a WYSIWYG (i.e., What You See Is What You Get) approach: Attention-based Self-guided Automatic Unit Test GenERation (AUGER), which contains two stages: defect detection and error triggering. In the former stage, AUGER first detects the proneness of defects. Then, in the latter stage, it guides to generate unit tests for triggering such an error with the help of critical information obtained by the former stage. To evaluate the effectiveness of AUGER, we conduct a large-scale experiment by comparing with the state-of-the-art (SOTA) approaches on the widely used datasets (i.e., Bears, Bugs.jar, and Defects4J). AUGER makes great improvements by 4.7% to 35.3% and 17.7% to 40.4% in terms of F1-score and Precision in defect detection, and can trigger 23 to 84 more errors than SOTAs in unit test generation. Besides, we also conduct a further study to verify the generalization in practical usage by collecting a new dataset from real-world projects.

Figures

Figures reproduced from arXiv: 2412.00828 by the authors.

Figure 1
Figure 1. Overview of AUGER tool. We encode the methods into token representations and input them into AUGER to detect whether there are defects. • Attention-guided Unit Test Generation. We conduct further analysis on the detected defective methods to identify the defective statements. Then, we guide the LLM to focus on the defective statements in order to generate unit tests that trigger the errors. • Unit Test Validation. W… view at source ↗
Figure 2
Figure 2. An example of prompt for LLM 2) Task 2: Prompt Preparation: AUGER is generalizable and can be extended to other programming languages by modifying the language-specific information in the prompt (e.g., [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Highlight attention for tokens in defective statement [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Venn diagram of AUGER and studied baselines [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Unique error triggered by AUGER Effectiveness of AUGER in real-world projects. Since the DeepSeek Coder’s pre-training data was sourced from GitHub prior to February 2023, in order to prevent data leakage, we collect defects from real-world projects starting from March…
Figure 6
Figure 6. Figure 6: The varying performance of AUGER and TOGA with different unit test candidate number on all defects Impact of unit test candidate number. According to the results in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The runtime overheads of baselines and AUGER [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation

    cs.SE 2026-04 conditional novelty 7.0 of 10

    SeGa extracts business semantics from requirements to generate unit tests that detect 22-25 more real-world business logic bugs than prior LLM-based methods in industrial Go projects.

  2. ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts

    cs.HC 2025-08 conditional novelty 6.0 of 10

    ViseGPT automatically converts user prompts into test cases and visualizes which steps of an LLM-generated data wrangling script pass or fail.

  3. VISCA: Inferring Component Abstractions for Automated End-to-End Testing

    cs.SE 2025-06 conditional novelty 6.0 of 10

    VISCA converts webpages into a semantic component abstraction and uses it as context for LLM-generated end-to-end tests, reporting 92% average feature coverage on E2EBench.

  4. Enhancing LLM's Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection

    cs.SE 2025-01 conditional novelty 6.0 of 10

    RATester injects gopls-fetched definitions into LLM prompts during unit test generation, achieving 26.25% average line coverage and more killed mutants than baselines.

  5. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

  6. Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

    cs.SE 2025-08 reject novelty 3.0 of 10

    A survey-plus-benchmark argues that current code LLMs fail safety thresholds and need stronger governance, but the supporting experiment is incomplete.

Reference graph

Works this paper leans on

72 extracted references · 47 canonical work pages · cited by 6 Pith papers

  1. [1]

    Deeplocalize: Fault localization for deep neural networks,

    M. Wardat, W. Le, and H. Rajan, “Deeplocalize: Fault localization for deep neural networks,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 251–262

    work page 2021

  2. [2]

    Modeling and discovering vulnerabilities with code property graphs,

    F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discovering vulnerabilities with code property graphs,” in 2014 IEEE Symposium on Security and Privacy . IEEE, 2014, pp. 590–604

    work page 2014

  3. [3]

    Vuldeepecker: A deep learning-based system for vulnerability detection,

    Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y . Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in Proceedings of the 25th Annual Network and Distributed System Security Symposium, 2018

    work page 2018

  4. [4]

    Vuldeelocator: a deep learning-based fine-grained vulnerability detector,

    Z. Li, D. Zou, S. Xu, Z. Chen, Y . Zhu, and H. Jin, “Vuldeelocator: a deep learning-based fine-grained vulnerability detector,” IEEE Transactions on Dependable and Secure Computing , 2021

    work page 2021

  5. [5]

    Large language models for test-free fault localization,

    A. Z. Yang, C. Le Goues, R. Martins, and V . Hellendoorn, “Large language models for test-free fault localization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    work page 2024

  6. [6]

    Cobot: static c/c++ bug detection in the presence of incomplete code,

    Q. Gao, S. Ma, S. Shao, Y . Sui, G. Zhao, L. Ma, X. Ma, F. Duan, X. Deng, S. Zhang et al. , “Cobot: static c/c++ bug detection in the presence of incomplete code,” in Proceedings of the 26th Conference on Program Comprehension, 2018, pp. 385–388

    work page 2018

  7. [7]

    Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization,

    X. Li, W. Li, Y . Zhang, and L. Zhang, “Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization,” in Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, 2019, pp. 169–180

    work page 2019

  8. [8]

    Multitask-based evaluation of open- source llm on software vulnerability,

    X. Yin, C. Ni, and S. Wang, “Multitask-based evaluation of open- source llm on software vulnerability,” IEEE Transactions on Software Engineering, 2024

    2024

Show all 72 references
  1. [9]

    Linevul: A transformer-based line- level vulnerability prediction,

    M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line- level vulnerability prediction,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 608–620

    2022

  2. [10]

    Devign: Effective vulnerabil- ity identification by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow, X. Du, and Y . Liu, “Devign: Effective vulnerabil- ity identification by learning comprehensive program semantics via graph neural networks,” in In Proceedings of the 33rd International Conference on Neural Information Processing Systems , 2019, p. 1...

    2019

  3. [11]

    Linevd: statement-level vulnerability detection using graph neural networks,

    D. Hin, A. Kan, H. Chen, and M. A. Babar, “Linevd: statement-level vulnerability detection using graph neural networks,” in Proceedings of the 19th international conference on mining software repositories , 2022, pp. 596–607

    2022

  4. [12]

    Vulnerability detection with fine- grained interpretations,

    Y . Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine- grained interpretations,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 292–303

    2021

  5. [13]

    Deep learning based vulnerability detection: Are we there yet,

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet,” IEEE Transactions on Software Engineering, 2021

    2021

  6. [14]

    Boosting coverage-based fault localization via graph-based representation learning,

    Y . Lou, Q. Zhu, J. Dong, X. Li, Z. Sun, D. Hao, L. Zhang, and L. Zhang, “Boosting coverage-based fault localization via graph-based representation learning,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundati...

    2021

  7. [15]

    Fault localization to detect co- change fixing locations,

    Y . Li, S. Wang, and T. N. Nguyen, “Fault localization to detect co- change fixing locations,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 659–671

    2022

  8. [16]

    An empirical study of deep learning models for vulnerability detection,

    B. Steenhoek, M. M. Rahman, R. Jiles, and W. Le, “An empirical study of deep learning models for vulnerability detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2237–2248

    2023

  9. [17]

    An industrial evaluation of unit test generation: Finding real faults in a financial application,

    M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track ...

    2017

  10. [18]

    Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t),

    S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t),” in2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IE...

    2015

  11. [19]

    A3test: Assertion-augmented automated test case generation,

    S. Alagarsamy, C. Tantithamthavorn, and A. Aleti, “A3test: Assertion-augmented automated test case generation,” arXiv preprint arXiv:2302.10352, 2023

    2023 arXiv

  12. [20]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” arXiv preprint arXiv:2009.05617, 2020

    2009 arXiv

  13. [21]

    Toga: A neural method for test oracle generation,

    E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, “Toga: A neural method for test oracle generation,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2130–2141

    2022

  14. [22]

    Meth- ods2test: A dataset of focal methods mapped to test cases,

    M. Tufano, S. K. Deng, N. Sundaresan, and A. Svyatkovskiy, “Meth- ods2test: A dataset of focal methods mapped to test cases,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 299–303

    2022

  15. [23]

    Bears: An extensible java bug benchmark for automatic program repair studies,

    F. Madeiral, S. Urli, M. Maia, and M. Monperrus, “Bears: An extensible java bug benchmark for automatic program repair studies,” in 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2019, pp. 468–478

    2019

  16. [24]

    Bugs. jar: A large-scale, diverse dataset of real-world java bugs,

    R. K. Saha, Y . Lyu, W. Lam, H. Yoshida, and M. R. Prasad, “Bugs. jar: A large-scale, diverse dataset of real-world java bugs,” in Proceedings of the 15th international conference on mining software repositories , 2018, pp. 10–13

    2018

  17. [25]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  18. [26]

    Replication,

    “Replication,” 2024. [Online]. Available: https://github.com/vinci-grape/ AUGER

    2024

  19. [27]

    Learning-based models for vulnerability detection: An extensive study,

    C. Ni, L. Shen, X. Xu, X. Yin, and S. Wang, “Learning-based models for vulnerability detection: An extensive study,” arXiv preprint arXiv:2408.07526, 2024

    2024 arXiv

  20. [28]

    Distinguishing look-alike innocent and vulnerable code by subtle semantic representation learning and explanation,

    C. Ni, X. Yin, K. Yang, D. Zhao, Z. Xing, and X. Xia, “Distinguishing look-alike innocent and vulnerable code by subtle semantic representation learning and explanation,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundat...

    2023

  21. [29]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 7212–7225

    2022

  22. [30]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , 2020, pp. 1536–1547

    2020

  23. [31]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, 2011, pp. 416–419

    2011

  24. [32]

    Randoop: feedback-directed random testing for java,

    C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” inCompanion to the 22nd ACM SIGPLAN conference on Object- oriented programming systems and applications companion , 2007, pp. 815–816

    2007

  25. [33]

    Analysing the fitness landscape of search-based software testing problems,

    A. Aleti, I. Moser, and L. Grunske, “Analysing the fitness landscape of search-based software testing problems,” Automated Software Engineer- ing, vol. 24, pp. 603–621, 2017

    2017

  26. [34]

    Mapping the effectiveness of automated test suite generation techniques,

    C. Oliveira, A. Aleti, L. Grunske, and K. Smith-Miles, “Mapping the effectiveness of automated test suite generation techniques,” IEEE Transactions on Reliability , vol. 67, no. 3, pp. 771–785, 2018

    2018

  27. [35]

    Reformulating branch coverage as a many-objective optimization problem,

    A. Panichella, F. M. Kifetew, and P. Tonella, “Reformulating branch coverage as a many-objective optimization problem,” in 2015 IEEE 8th international conference on software testing, verification and validation (ICST). IEEE, 2015, pp. 1–10

    2015

  28. [36]

    Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,

    ——, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Transactions on Software Engineering, vol. 44, no. 2, pp. 122–158, 2017

    2017

  29. [37]

    Deepseek coder: Let the code write itself,

    D. AI, “Deepseek coder: Let the code write itself,” https://github.com/ deepseek-ai/DeepSeek-Coder, 2023

    2023

  30. [38]

    Code llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  31. [39]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  32. [40]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    2023 arXiv

  33. [41]

    Javaparser,

    “Javaparser,” 2024. [Online]. Available: https://javaparser.org/

    2024

  34. [42]

    Enhancing document-level event argument extraction with contextual clues and role relevance,

    W. Liu, S. Cheng, D. Zeng, and H. Qu, “Enhancing document-level event argument extraction with contextual clues and role relevance,” arXiv preprint arXiv:2310.05991, 2023

    2023 arXiv

  35. [43]

    Beyond single-event extraction: Towards efficient document-level multi-event argument extraction,

    W. Liu, L. Zhou, D. Zeng, Y . Xiao, S. Cheng, C. Zhang, G. Lee, M. Zhang, and W. Chen, “Beyond single-event extraction: Towards efficient document-level multi-event argument extraction,” arXiv preprint arXiv:2405.01884, 2024

    2024 arXiv

  36. [44]

    Adversarial training methods for semi-supervised text classification,

    T. Miyato, A. M. Dai, and I. Goodfellow, “Adversarial training methods for semi-supervised text classification,” arXiv preprint arXiv:1605.07725, 2016

    2016 arXiv

  37. [45]

    R-drop: Regularized dropout for neural networks,

    L. Wu, J. Li, Y . Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, T.-Y . Liu et al., “R-drop: Regularized dropout for neural networks,” Advances in Neural Information Processing Systems , vol. 34, pp. 10 890–10 905, 2021

    2021

  38. [46]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821 , 2021

    2021 arXiv

  39. [47]

    Enhancing discriminative tasks by guiding the pre-trained language model with large language model’s experience,

    X. Yin, C. Ni, X. Xu, X. Li, and X. Yang, “Enhancing discriminative tasks by guiding the pre-trained language model with large language model’s experience,” arXiv preprint arXiv:2408.08553 , 2024

    2024 arXiv

  40. [48]

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,

    Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023, 2023

    2023 arXiv

  41. [49]

    Rectifier: Code translation with corrector via llms,

    X. Yin, C. Ni, T. N. Nguyen, S. Wang, and X. Yang, “Rectifier: Code translation with corrector via llms,” arXiv preprint arXiv:2407.07472 , 2024

    2024 arXiv

  42. [50]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  43. [51]

    Thinkrepair: Self-directed automated program repair,

    X. Yin, C. Ni, S. Wang, Z. Li, L. Zeng, and X. Yang, “Thinkrepair: Self-directed automated program repair,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1274–1286

    2024

  44. [52]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  45. [53]

    An extensive study on pre-trained models for program understanding and generation,

    Z. Zeng, H. Tan, H. Zhang, J. Li, Y . Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis , 2022, pp. 39–51

    2022

  46. [54]

    What do they capture? a structural analysis of pre-trained language models for source code,

    Y . Wan, W. Zhao, H. Zhang, Y . Sui, G. Xu, and H. Jin, “What do they capture? a structural analysis of pre-trained language models for source code,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 2377–2388

    2022

  47. [55]

    Towards more realistic evaluation for neural test oracle generation,

    Z. Liu, K. Liu, X. Xia, and X. Yang, “Towards more realistic evaluation for neural test oracle generation,” in Proceedings of the 32th International Symposium on Software Testing and Analysis . ACM, 2023

    2023

  48. [56]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” arXiv preprint arXiv:2305.07922 , 2023

    2023 arXiv

  49. [57]

    A few billion lines of code later: using static analysis to find bugs in the real world,

    A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri- Gros, A. Kamsky, S. McPeak, and D. Engler, “A few billion lines of code later: using static analysis to find bugs in the real world,” Communications of the ACM , vol. 53, no. 2, pp. 66–75, 2010

    2010

  50. [58]

    Why don’t software developers use static analysis tools to find bugs?

    B. Johnson, Y . Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 2013, pp. 672–681

    2013

  51. [59]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-pe...

    2019

  52. [60]

    Hugging face,

    “Hugging face,” 2023. [Online]. Available: https://huggingface.co

    2023

  53. [61]

    Defect identification, categorization, and repair: Better together,

    C. Ni, K. Yang, X. Xia, D. Lo, X. Chen, and X. Yang, “Defect identification, categorization, and repair: Better together,” arXiv preprint arXiv:2204.04856, 2022

    2022 arXiv

  54. [62]

    Data mining static code attributes to learn defect predictors,

    T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” IEEE transactions on software engineering, vol. 33, no. 1, pp. 2–13, 2006

    2006

  55. [63]

    Predicting defects for eclipse,

    T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for eclipse,” in Third International Workshop on Predictor Models in Software Engineering (PROMISE’07: ICSE Workshops 2007) . IEEE, 2007, pp. 9–9

    2007

  56. [64]

    The influence of organizational structure on software quality: an empirical case study,

    N. Nagappan, B. Murphy, and V . Basili, “The influence of organizational structure on software quality: an empirical case study,” in Proceedings of the 30th international conference on Software engineering , 2008, pp. 521–530

    2008

  57. [65]

    Use of relative code churn measures to predict system defect density,

    N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in Proceedings of the 27th international conference on Software engineering . ACM, 2005, pp. 284–292

    2005

  58. [66]

    Predicting fault incidence using software change history,

    T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy, “Predicting fault incidence using software change history,” IEEE Transactions on software engineering, vol. 26, no. 7, pp. 653–661, 2000

    2000

  59. [67]

    Predicting faults from cached history,

    S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller, “Predicting faults from cached history,” in 29th International Conference on Software Engineering (ICSE’07). IEEE, 2007, pp. 489–498

    2007

  60. [68]

    The best of both worlds: integrating semantic features with expert features for defect prediction and localization,

    C. Ni, W. Wang, K. Yang, X. Xia, K. Liu, and D. Lo, “The best of both worlds: integrating semantic features with expert features for defect prediction and localization,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundati...

    2022

  61. [69]

    A systematic survey of just-in-time software defect prediction,

    Y . Zhao, K. Damevski, and H. Chen, “A systematic survey of just-in-time software defect prediction,” ACM Computing Surveys , vol. 55, no. 10, pp. 1–35, 2023

    2023

  62. [70]

    Studying high impact fix- inducing changes,

    A. T. Misirli, E. Shihab, and Y . Kamei, “Studying high impact fix- inducing changes,” Empirical Software Engineering, vol. 21, pp. 605–641, 2016

    2016

  63. [71]

    Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction,

    C. Pornprasit and C. K. Tantithamthavorn, “Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . IEEE, 2021, pp. 369–379

    2021

  64. [72]

    Pyexplainer: Explaining the predictions of just-in-time defect models,

    C. Pornprasit, C. Tantithamthavorn, J. Jiarpakdee, M. Fu, and P. Thong- tanunam, “Pyexplainer: Explaining the predictions of just-in-time defect models,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 407–418

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.