Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning

Charaka Geethal Kapugama

REVIEW 5 major objections 5 minor 33 references

Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ISONOISE isolates human mislabeled test cases with over 67% accuracy.

desk verdict ISONOISE is a plausible new noisy-label detector for tiny HIOL training sets; the idea is worth a serious referee, but the evaluation overstates precision and the main loop has a bug. read the letter →

arxiv 2506.13273 v1 pith:E4NVS6CZ submitted 2025-06-16 cs.SE

Charaka Geethal Kapugama This is my paper · ORCID

classification cs.SE

keywords noisylabeldetectionhuman-in-the-looporaclelearningtestautomationactivesoftwaretestingnoiseautomaticprogramrepair

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that human labeling mistakes in human-in-the-loop oracle learning can be found automatically, without any extra labeled data. It introduces ISONOISE, which scores each test case by how often a family of classifiers trained on fuzzed variants disagrees with the human's label. Test cases with high disagreement are presented to the human for relabeling, and the oracle is retrained whenever a mistake is confirmed. On 552 buggy programs from Codeflaws, ISONOISE identifies the mislabeled test cases with over 67% accuracy for most subjects, while sending only a handful of relabeling queries. If true, this makes human-in-the-loop oracle learning and the program repair built on it more robust to noisy human labels.

What carries the argument

The load-bearing mechanism is the disagreement score from Algorithm 2, adapted from Bougelia et al.'s agree-to-disagree idea. For a test case $t$, the algorithm excludes $t$, fuzzes it $N$ times to produce mutants, labels each mutant with the existing oracle $O$, trains a new classifier on the suite plus the mutant, and increments disagreement whenever that classifier's prediction for $t$ differs from the human's label. This score separates suspicious test cases from trustworthy ones; then an intermediate classifier $O_N$ trained only on the low-disagreement cases decides which suspicious cases to send for relabeling. The threshold $D$ (set to 15 in the experiments) and the retraining loop are what make the method work with only 20--50 training examples.

What would settle it

Take a Codeflaws subject, invert 30% of the labels randomly, and deliberately use an oracle O that has been trained only on the inverted labels so that its accuracy is near chance; if ISONOISE's detection accuracy drops to roughly 50% while it still sends many relabeling queries, the disagreement calculation is too dependent on the corrupted oracle to be reliable.

Watch

Extended reading notes

Core claim

The central claim is that noisy labels introduced by the human during LEARN2FIX-style oracle learning can be isolated using only the already-trained automatic oracle and its training suite. ISONOISE computes a disagreement score for each test case by repeatedly mutating it through arithmetic fuzzing, labelling the mutant with the existing oracle, retraining a fresh classifier on the suite with the mutant added, and checking whether that classifier disagrees with the human's label. Test cases whose disagreement score exceeds a threshold are set aside, an intermediate oracle is trained on the remaining cases, and only the set-aside cases are candidates for relabeling. When a relabeling confirms a mistake, the oracle is retrained and the whole process restarts. The paper reports that for the majority of 552 Codeflaws subjects, the method detects the mislabeled cases with accuracy above 67%, with median accuracy above 70% for failing tests mislabeled as passing and above 60% for passing tests mislabeled as failing.

Load-bearing premise

The disagreement scores keep working even when the oracle used to label the fuzzed mutants is itself partly corrupted by the very mislabeled tests we are trying to find.

Editorial extensions

If this is right

If ISONOISE works as reported, human-in-the-loop oracle learning no longer has to assume perfect labeling; it can detect and correct a large share of human mistakes.
Since failing tests mislabeled as passing are found at median accuracy above 70%, the failure condition learned by LEARN2FIX is less likely to be silently corrupted.
The relabeling burden is small: the median number of relabeling queries is only slightly above the number of actual mislabeled tests, and the probability that a query hits a mislabeled test is far above random.
The same disagreement-driven isolation could be retrofitted to other oracle-learning and program-repair pipelines that suffer from label noise.
Accuracy degrades as the noise threshold rises, so the method is most useful when the human error rate is moderate (5--20%).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The disagreement-score approach might transfer to oracle learning for string inputs if fuzzing mutants are generated by grammar-based mutations, though the paper leaves this to future work.
The sensitivity of the disagreement threshold $D$ is likely the main practical tuning point; a data-driven way to set $D$ from the distribution of scores could make ISONOISE more robust across subjects.
Because detection accuracy falls as label noise increases, one could add an explicit early stop: if the intermediate oracle's confidence on relabeled cases is low, stop rather than risk confirming wrong labels.
The paper's 67% benchmark suggests a practical upper bound on how much label noise human-in-the-loop oracle learning can tolerate and still be corrected; estimating that bound per subject could guide when to invest in a reference implementation for relabeling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: this paper introduces ISONOISE, a technique to spot mislabelled test cases in human-in-the-loop oracle learning (HIOL) when the training suite is only 20–50 cases and there is no unlabelled pool. That setting is real, and as far as I can tell ISONOISE is the first method aimed at it. The core trick is to add fuzzed mutants labelled by the current oracle to the training set, train an intermediate classifier, and measure disagreement with the original human label. It borrows deliberately from Bougelia et al. and BDAL, and the iterative relabelling loop is a reasonable adaptation.

What the paper does well: it runs on 552 Codeflaws subjects, reports detection accuracy and relabelling effort, and the code and data are on GitHub. It also openly admits that a compromised oracle degrades the disagreement calculation (Section V.A), which is the main conceptual risk. That honesty counts.

The soft spots are mostly in the measurement. 'Accuracy' here is recall-like; there is no precision or false-alarm rate, so we do not know how many relabelling queries are wasted. There is no comparison against a random-query baseline or an existing noisy-label method, and the disagreement threshold D=15 is tuned on pilot data with no sensitivity analysis. The box plots have no confidence intervals. Those are fixable, but they matter because the headline 'over 67%' is not the full picture.

The more serious issue is Algorithm 1. If no test case in TS meets the relabel condition, NoNoiseFound stays False and the while loop never terminates. And when one relabelled test case comes back with the same label, the loop stops and ignores the rest of TS. That is a termination bug, not a stylistic quibble, and it needs fixing before the method can be called a procedure. The 30% noisy threshold in Fig. 2b also contradicts the 5/10/20% setup.

The circular-oracle worry from the stress test is real but already disclosed; the paper should quantify how detection varies with oracle quality instead of just admitting the dependence. I would not call it a fatal flaw, because the external benchmark and golden labels break the circularity of the evaluation itself.

Bottom line: the paper deserves a serious referee and major revision. It is a niche but legitimate contribution to software testing, and the author is thinking clearly about the problem. For my own work I would probably not cite it in the next year, but I would bring it to a reading group focused on oracle learning.

Referee Report

5 major / 5 minor

Summary. The paper introduces ISONOISE, a technique to detect mislabelled test cases in human-in-the-loop oracle learning (HIOL) for programs with numeric inputs. Given a compromised automatic test oracle O and its training suite T, ISONOISE computes per-test disagreement scores by repeatedly training intermediate classifiers on fuzzed mutants labelled by O (Algorithm 2), thresholds these scores to form a suspicious set TS, trains a cleaner intermediate oracle ON on the remaining cases, and then uses ON's predictions to decide which suspicious cases to send for human relabelling, retraining O whenever a noisy label is found (Algorithm 1). The approach is evaluated on 552 Codeflaws subjects within the LEARN2FIX framework at 5%, 10%, and 20% injected label noise, reporting detection accuracy above 67% for a majority of subjects and a small number of relabelling queries. The authors also provide a public repository with implementation and data.

Significance. If the claims hold, ISONOISE would be a useful contribution: it addresses a real problem in HIOL, works with very small training suites (around 20-50 cases), and does not require an additional labelled dataset. The paper has concrete strengths: it builds on an existing method (Bouguelia et al.), evaluates on a substantial benchmark (552 subjects, 30 repetitions per subject), and ships reproducibility artifacts. However, the evaluation currently measures only recall-like accuracy, omits precision and false-alarm costs, compares against no noisy-label baseline, and leaves the dependence on the corrupted oracle's quality unquantified. The central idea is defensible, but the evidence as presented does not yet support the strength of the claimed practical benefit.

major comments (5)

[Section V-A, Fig. 1] The 'detection accuracy' metric is effectively recall: the fraction of actually mislabelled test cases that ISONOISE flags. Precision and false-positive rate are never reported, so a reader cannot tell how many relabelling queries are wasted on correctly labelled cases. This is load-bearing because RQ.2's 'small number of relabelling queries' claim depends on both the number of queries and their utility; a method that sends every case for relabelling would achieve 100% recall but be useless. Please report a confusion-matrix breakdown (true positives, false positives, false negatives) or at least precision and false-alarm rate per noisy-label threshold, and define the accuracy metric explicitly in Section IV-C.
[Section III-A, Algorithm 2, Line 6] The disagreement signal relies on labels assigned by the already-compromised oracle O to fuzzed mutants (t' labelled by O). When O is corrupted, the disagreement score measures agreement with a potentially wrong label, not with ground truth. The paper acknowledges this in Section V.A ('A highly compromised oracle adversely affects the disagreement calculation process in Algorithm 2'), but it never quantifies how ISONOISE's recall degrades as a function of O's accuracy. This is precisely the regime the technique is meant to address. Please add a controlled study that varies O's quality (e.g., by injecting different levels of noise before running ISONOISE, or by corrupting O directly) and plots detection recall and precision against O accuracy, so the operating envelope of Algorithm 2 is explicit.
[Section IV-C and Section V-B] The experimental setup lists only 5%, 10%, and 20% noisy-label thresholds (Section IV-C: 'The above process is repeated for the 5%, 10% and 20% noisy label thresholds.'), and Fig. 2a/2b plot only these three thresholds. Yet Section V.B states 'In the 30% noisy label threshold, this probability exceeds 0.5' and later compares against a random baseline probability of 0.3 at 30%. A 30% condition appears neither in the setup nor in the figures, making the reported result unreproducible. Please either add the 30% threshold to the experiments and figures, or remove the unsubstantiated 30% references and any conclusions drawn from them.
[Section IV-C, Disagreement Threshold, and Section V] The disagreement threshold D=15 is the key control for separating 'highly disagreeing' from 'slightly disagreeing' test cases in Algorithm 1, yet it is set based only on unspecified pilot experiments and no sensitivity analysis is provided. The conclusions in Section V could depend critically on this single value. Please report how detection accuracy and relabelling queries vary as D changes (e.g., a sweep over a range around 15), and similarly for the fuzzing iteration count N=20, so the reader can judge the robustness of the reported results.
[Section V, RQ.2] The paper claims that ISONOISE 'significantly increases the likelihood of sending an incorrectly labelled test for relabelling' (Introduction contribution 3 and Section V.B Result), but no statistical significance tests, confidence intervals, or effect-size estimates are provided; Fig. 1 and Fig. 2 are box plots without error bars or significance annotations. Given that each experiment is repeated 30 times, the data are sufficient to report standard errors or paired comparisons against a random relabelling baseline with the same query budget. Please add such statistical support, otherwise the word 'significantly' is unsupported.

minor comments (5)

[Introduction, first paragraph of the contribution description] There is a typo 'incorreclty' that should read 'incorrectly'.
[Algorithm 1, Line 16] Line 16 says 'Let hold be the previous human label of tn', but the loop variable is ts, not tn; this makes the pseudocode inconsistent with the surrounding text.
[Algorithm 1, Line 21] The sentence 'Change the label of to hnew' is missing the object; it should read 'Change the label of ts to hnew'.
[Fig. 1 and Fig. 2 captions] The captions do not define 'Failing-Incorrect' and 'Passing-Incorrect', nor do they state whether the plotted values are medians or means across subjects; please add explicit definitions and clarify the aggregation procedure.
[Section IV-C, 'Disagreement Threshold'] The text says 'The Disagreement Threshold was decided based on some pilot experiments' but gives no details about these pilots (number of subjects, range of values tried, criterion for selection). A sentence or two describing the pilot procedure would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: evaluation uses external Codeflaws ground truth; oracle-dependent disagreement is an acknowledged limitation, not a circular derivation.

full rationale

The derivation is self-contained. ISONOISE's central claim—identifying mislabelled test cases—is evaluated against ground truth produced by comparing buggy and golden Codeflaws programs, i.e., an external benchmark, not against its own inputs. The human oracle is simulated from the same golden outputs, and the relabelling process is assumed correct. The disagreement signal in Algorithm 2 does use the compromised oracle O to label fuzzed mutants, which the paper itself acknowledges in Section V.A: 'A highly compromised oracle adversely affects the disagreement calculation process in Algorithm 2, making it difficult to trace incorrectly labelled test cases.' That is a robustness limitation of the signal, not a circular derivation: the detection target is the human's label error, and O's predictions are not the final evaluation metric. The self-citations to LEARN2FIX supply the experimental setup and background facts, but no load-bearing step reduces to them. The disagreement threshold D was fixed by pilot experiments rather than fitted to the evaluation outcomes. The only notable reporting inconsistency, a mention of a '30% noisy label threshold' in RQ.2 while the setup lists 5%, 10%, and 20%, is a presentation issue and does not affect the circularity assessment.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger captures the small number of parameters and assumptions behind ISONOISE. D and N are the main hand-chosen parameters; D was selected through pilot experiments and directly controls which cases are considered suspicious. The method assumes the initial failing test is correct, that the compromised oracle can still label mutated inputs usefully, and that relabelling answers are trusted. The evaluation adds a fixed-length numeric input assumption that is broader in the abstract.

free parameters (2)

Disagreement threshold D = 15
Section IV-C states the Disagreement Threshold was decided based on some pilot experiments. This controls which test cases are flagged as suspicious, and no sensitivity analysis is reported.
Fuzzing iterations N = 20
Section IV-C sets fuzzing iterations to 20. This is a hand-chosen parameter affecting the stability of disagreement scores.

assumptions (4)

domain assumption The initial failing test f is correctly labelled.
Algorithm 1 Line 4 excludes f from suspicion and uses it as a trusted seed; if f is mislabelled, the intermediate oracle is trained on a wrong anchor.
domain assumption The compromised oracle O's predictions on fuzzed mutants are accurate enough to yield useful disagreement signals.
Algorithm 2 Lines 5-8 labels each mutant t' with O(t') and trains OT on it; if O is heavily corrupted, disagreement scores may not distinguish noise.
domain assumption Human relabelling answers are correct.
Section IV-C states relabelling queries are assumed correctly answered; errors here would propagate into the retrained oracle.
domain assumption Test inputs have fixed length and numeric outputs.
Section IV-B criterion 5 selects subjects meeting this; the abstract says numeric inputs generally, but the implementation and decision-tree classifiers require the fixed-length form.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning." pith.science (2026). https://pith.science/paper/E4NVS6CZ

@misc{pith2026250613273,
  author       = {Pith},
  title        = {Pith review of: Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4NVS6CZ}},
  note         = {Machine review of arXiv:2506.13273}
}

read the original abstract

Incorrectly labelled test cases can adversely affect the training process of human-in-the-loop oracle learning tech-niques. This paper introduces ISONOISE, a technique designed to identify such mislabelled test cases introduced during human-in-the-loop oracle learning. This technique can be applied to programs taking numeric inputs. Given a compromised automatic test oracle and its training test suite, ISONOISE first isolates thetest cases suspected of being mislabelled. This task is performed based on the level of disagreement of a test case with respect to the others. An intermediate automatic test oracle is trained based on the slightly disagreeing test cases. Based on the predictions of this intermediate oracle, the test cases suspected of being mislabelled are systematically presented for relabelling. When mislabelled test cases are found, the intermediate test oracle is updated. This process repeats until no mislabelled test case is found in relabelling. ISONOISE was evaluated within the human-in-the-loop oracle learning method used in LEARN2FIX. Experimental results demonstrate that ISONOISE can identify mislabelled test cases introduced by the human in LEARN2FIX with over 67% accuracy, while requiring only a small number of relabelling queries. These findings highlight the potential of ISONOISE to enhance the reliability of human-in-the-loop oracle learning.

Figures

Figures reproduced from arXiv: 2506.13273 by the authors.

**Figure 2.** Relabelling Effort Interpretation. The results suggest that ISONOISE uses a small number of relabelling queries to identify incorrectly labelled test cases and to rectify automatic test oracles in most subjects. The lower relabelling effort suggests that ISONOISE effectively reduces the search space needed to examine in relabelling. When the entire allocated query budget is used by LEARN2FIX, the probability of rand… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 26 canonical work pages

[1]

On testing non-testable programs,

E. J. Weyuker, “On testing non-testable programs,” The Computer Journal, vol. 25, no. 4, pp. 465–470, 1982

work page 1982
[2]

The oracle problem in software testing: A survey,

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

2014
[3]

Novel applications of machine learning in software test- ing,

L. C. Briand, “Novel applications of machine learning in software test- ing,” in 2008 The Eighth International Conference on Quality Software . IEEE, 2008, pp. 3–10

work page 2008
[4]

Bug characteris- tics in open source software,

L. Tan, C. Liu, Z. Li, X. Wang, Y . Zhou, and C. Zhai, “Bug characteris- tics in open source software,” Empirical Software Engineering, vol. 19, no. 6, pp. 1665–1705, 2014

work page 2014
[5]

Human-in-the-loop automatic program repair,

C. Geethal, M. B ¨ohme, and V .-T. Pham, “Human-in-the-loop automatic program repair,” IEEE Transactions on Software Engineering , 2023

work page 2023
[6]

Human-in-the-loop automatic program repair,

M. B ¨ohme, C. Geethal, and V .-T. Pham, “Human-in-the-loop automatic program repair,” in2020 IEEE 13th international conference on software testing, validation and verification (ICST) . IEEE, 2020, pp. 274–285

work page 2020
[7]

Human-in-the-loop oracle learning for semantic bugs in string processing programs,

C. Geethal, V .-T. Pham, A. Aleti, and M. B ¨ohme, “Human-in-the-loop oracle learning for semantic bugs in string processing programs,” in Symposium on Software Testing and Analysis (ISSTA’22) , 2022

work page 2022
[8]

Agreeing to disagree: Active learning with noisy labels without crowdsourcing,

M.-R. Bouguelia, S. Nowaczyk, K. Santosh, and A. Verikas, “Agreeing to disagree: Active learning with noisy labels without crowdsourcing,” International journal of machine learning and cybernetics , vol. 9, pp. 1307–1319, 2018

work page 2018

Show all 33 references

[9]

Codeflaws: A programming competition benchmark for evaluating automated pro- gram repair tools,

S. H. Tan, J. Yi, Yulis, S. Mechtaev, and A. Roychoudhury, “Codeflaws: A programming competition benchmark for evaluating automated pro- gram repair tools,” in 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) , 2017, pp. 180–182

2017
[10]

Active learning literature survey,

B. Settles, “Active learning literature survey,” University of Wisconsin- Madison Department of Computer Sciences, Tech. Rep., 2009

2009
[11]

Fuzzing: State of the art,

H. Liang, X. Pei, X. Jia, W. Shen, and J. Zhang, “Fuzzing: State of the art,” IEEE Transactions on Reliability , vol. 67, no. 3, pp. 1199–1218, 2018

2018
[12]

Bidirectional active learning: A two-way exploration into unlabeled and labeled data set,

X.-Y . Zhang, S. Wang, and X. Yun, “Bidirectional active learning: A two-way exploration into unlabeled and labeled data set,” IEEE transactions on neural networks and learning systems , vol. 26, no. 12, pp. 3034–3044, 2015

2015
[13]

Confident learning: Estimating uncertainty in dataset labels,

C. Northcutt, L. Jiang, and I. Chuang, “Confident learning: Estimating uncertainty in dataset labels,” Journal of Artificial Intelligence Research, vol. 70, pp. 1373–1411, 2021

2021
[14]

Qactor: Active learning on noisy labels,

T. Younesian, Z. Zhao, A. Ghiassi, R. Birke, and L. Y . Chen, “Qactor: Active learning on noisy labels,” in Asian Conference on Machine Learning. PMLR, 2021, pp. 548–563

2021
[15]

Stream-based active learning in the presence of label noise,

M.-R. Bouguelia, Y . Bela ¨ıd, and A. Bela ¨ıd, “Stream-based active learning in the presence of label noise,” in 4th International confer- ence on pattern recognition applications and methods-ICPRAM 2015 . SciTePress, 2015, pp. 25–34

2015
[16]

Active learning for noisy data streams using weak and strong labelers,

T. Younesian, D. Epema, and L. Y . Chen, “Active learning for noisy data streams using weak and strong labelers,” arXiv preprint arXiv:2010.14149, 2020

2010 arXiv
[17]

Active learning for data streams: a survey,

D. Cacciarelli and M. Kulahci, “Active learning for data streams: a survey,” Machine Learning, vol. 113, no. 1, pp. 185–239, 2024

2024
[18]

Boostclean: Automated error detection and repair for machine learning,

S. Krishnan, M. J. Franklin, K. Goldberg, and E. Wu, “Boostclean: Automated error detection and repair for machine learning,” arXiv preprint arXiv:1711.01299, 2017

2017 arXiv
[19]

B-repair: repairing b-models using machine learning,

C.-H. Cai, J. Sun, and G. Dobbie, “B-repair: repairing b-models using machine learning,” in 2018 23rd International Conference on Engineer- ing of Complex Computer Systems (ICECCS) . IEEE, 2018, pp. 31–40

2018
[20]

Hybridrepair: towards annotation-efficient repair for deep learning models,

Y . Li, M. Chen, and Q. Xu, “Hybridrepair: towards annotation-efficient repair for deep learning models,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 227–238

2022
[21]

Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks,

Y . Feng, Q. Shi, X. Gao, J. Wan, C. Fang, and Z. Chen, “Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks,” in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2020, pp. 177–188

2020
[22]

Deep learning for visual understanding: A review,

Y . Guo, Y . Liu, A. Oerlemans, S. Lao, S. Wu, and M. S. Lew, “Deep learning for visual understanding: A review,” Neurocomputing, vol. 187, pp. 27–48, 2016

2016
[23]

Automatic model repair using reinforcement learning

A. Barriga, A. Rutle, and R. Heldal, “Automatic model repair using reinforcement learning.” in MoDELS (Workshops), 2018, pp. 781–786

2018
[24]

The art, science, and engineering of fuzzing: A survey,

V . J. Man `es, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo, “The art, science, and engineering of fuzzing: A survey,” IEEE Transactions on Software Engineering , vol. 47, no. 11, pp. 2312–2331, 2019

2019
[25]

A divide-and-conquer method for multi-net classifiers,

D. Frosyniotis, A. Stafylopatis, and A. Likas, “A divide-and-conquer method for multi-net classifiers,” Pattern Analysis & Applications, vol. 6, pp. 32–40, 2003

2003
[26]

Automatic software repair: A survey,

L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” IEEE Transactions on Software Engineering, vol. 45, no. 1, pp. 34–67, 2017

2017
[27]

Classification based on decision tree algorithm for machine learning,

B. Charbuty and A. Abdulazeez, “Classification based on decision tree algorithm for machine learning,” Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 20–28, 2021

2021
[28]

The ManyBugs and IntroClass benchmarks for automated repair of C programs,

C. Le Goues, N. Holtschulte, E. K. Smith, Y . Brun, P. Devanbu, S. Forrest, and W. Weimer, “The ManyBugs and IntroClass benchmarks for automated repair of C programs,” IEEE Transactions on Software Engineering (TSE) , vol. 41, no. 12, pp. 1236–1256, December 2015, DOI: 10.1109...

2015
[29]

Pair programming: what’s in it for me?

A. Begel and N. Nagappan, “Pair programming: what’s in it for me?” in Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement , 2008, pp. 120–128

2008
[30]

Semantic program repair using a reference implementation,

S. Mechtaev, M.-D. Nguyen, Y . Noller, L. Grunske, and A. Roychoud- hury, “Semantic program repair using a reference implementation,” in Proceedings of the 40th International Conference on Software Engineer- ing, 2018, pp. 129–139

2018
[31]

Angelix: Scalable multiline program patch synthesis via symbolic analysis,

S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” in2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) , May 2016, pp. 691–701

2016
[32]

Genprog: A generic method for automatic software repair,

C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,” Ieee transactions on software engineering, vol. 38, no. 1, pp. 54–72, 2011

2011
[33]

A survey on software fault localization,

W. E. Wong, R. Gao, Y . Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,”IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707–740, 2016

2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

On testing non-testable programs,

E. J. Weyuker, “On testing non-testable programs,” The Computer Journal, vol. 25, no. 4, pp. 465–470, 1982

work page 1982

[2] [2]

The oracle problem in software testing: A survey,

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

2014

[3] [3]

Novel applications of machine learning in software test- ing,

L. C. Briand, “Novel applications of machine learning in software test- ing,” in 2008 The Eighth International Conference on Quality Software . IEEE, 2008, pp. 3–10

work page 2008

[4] [4]

Bug characteris- tics in open source software,

L. Tan, C. Liu, Z. Li, X. Wang, Y . Zhou, and C. Zhai, “Bug characteris- tics in open source software,” Empirical Software Engineering, vol. 19, no. 6, pp. 1665–1705, 2014

work page 2014

[5] [5]

Human-in-the-loop automatic program repair,

C. Geethal, M. B ¨ohme, and V .-T. Pham, “Human-in-the-loop automatic program repair,” IEEE Transactions on Software Engineering , 2023

work page 2023

[6] [6]

Human-in-the-loop automatic program repair,

M. B ¨ohme, C. Geethal, and V .-T. Pham, “Human-in-the-loop automatic program repair,” in2020 IEEE 13th international conference on software testing, validation and verification (ICST) . IEEE, 2020, pp. 274–285

work page 2020

[7] [7]

Human-in-the-loop oracle learning for semantic bugs in string processing programs,

C. Geethal, V .-T. Pham, A. Aleti, and M. B ¨ohme, “Human-in-the-loop oracle learning for semantic bugs in string processing programs,” in Symposium on Software Testing and Analysis (ISSTA’22) , 2022

work page 2022

[8] [8]

Agreeing to disagree: Active learning with noisy labels without crowdsourcing,

M.-R. Bouguelia, S. Nowaczyk, K. Santosh, and A. Verikas, “Agreeing to disagree: Active learning with noisy labels without crowdsourcing,” International journal of machine learning and cybernetics , vol. 9, pp. 1307–1319, 2018

work page 2018