{"id":"b6e9e481-8b26-42e8-b45e-68c49371ac8d","arxiv_id":"2506.13273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ISONOISE detects mislabelled test cases in small human-in-the-loop oracle learning training sets by measuring disagreement with fuzzed variants and iteratively relabelling suspicious cases.","lead":"A new technique, ISONOISE, picks out wrongly labelled test cases produced during human-in-the-loop oracle learning for numeric-input programs. It uses disagreement scores between an automatic test oracle and fuzzed variants, then sends only suspicious cases for relabelling, reducing effort.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core disagreement signal uses the compromised oracle O to label fuzzed mutants, so ISONOISE may be unreliable precisely when noise is high; the paper acknowledges degradation but does not quantify the dependence on O's quality.","rationale":"The reader's weakest_assumption pointed to the reliability of disagreement scores computed via the compromised oracle, and I agree this is the most load-bearing condition. The method's entire premise is that disagreement scores can reveal label noise even when the only signal available is produced by the corrupted system. The paper provides a plausible heuristic, but the evaluation does not isolate this dependence: it reports aggregate recall-like accuracy across subjects without conditioning on the quality of O, and it lacks a baseline. The internal admission that a compromised oracle 'adversely affects' the disagreement calculation is itself evidence that the assumption is insecure. My proposed test directly measures the contribution of O's labels by replacing them with ground-truth labels; if the gap is large, the method is not robust to the very condition it is supposed to handle. I do not think this warrants rejection — the method may still work at low noise and with a fix to the termination issue — but it strengthens the reader's CONDITIONAL verdict. The metric/baseline issue is real but secondary; without addressing the core circularity, even a proper baseline would not establish the mechanism.","tokens_in":11029,"tokens_out":6631,"duration_ms":75214,"concrete_test":"Using the Codeflaws golden versions, re-run the evaluation twice: once with Algorithm 2 labelling each fuzzed mutant t' with O's prediction (current paper), and once with the true label from the golden version, holding all other parameters fixed (D=15, N=20). Compare ISONOISE's detection accuracy and relabelling queries in the two conditions. If the true-label version yields substantially higher recall, the method's dependence on O's reliability is confirmed. Additionally, compute O's own classification accuracy on the test suite after each noise threshold and plot ISONOISE recall per subject against O accuracy; if recall is near chance for subjects where O accuracy < 70%, the headline claim should be qualified to low-noise regimes only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 2 (CALCULATE DISAGREEMENT, Section III-A) trains an intermediate classifier OT on T' plus a fuzzed mutant t' labelled by the already-compromised oracle O (Line 6). If O systematically mislabels test cases, the label L it assigns to t' is also suspect, and the disagreement score for t measures agreement with a possibly incorrect label, not with ground truth. This is a circular use of the oracle to detect the oracle's own errors. The paper's own Section V.A admits that 'a highly compromised oracle adversely affects the disagreement calculation process in Algorithm 2, making it difficult to trace incorrectly labelled test cases.' Yet the central claim asserts 'over 67% accuracy' across the tested 5%, 10%, and 20% noisy-label thresholds, with no analysis of how ISONOISE's recall varies with the quality of O. At 20% noise, O may already be too corrupted for the disagreement scores to separate noisy from correct labels, and the box plots in Fig. 1 do not show the conditional dependence on O accuracy. The internal text also mentions a '30% noisy label threshold' in RQ.2 while the experimental setup lists only 5%, 10%, and 20%, adding uncertainty about the claimed operating range.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ISONOISE, a technique to detect mislabelled test cases in human-in-the-loop oracle learning (HIOL) for programs with numeric inputs. Given a compromised automatic test oracle O and its training suite T, ISONOISE computes per-test disagreement scores by repeatedly training intermediate classifiers on fuzzed mutants labelled by O (Algorithm 2), thresholds these scores to form a suspicious set TS, trains a cleaner intermediate oracle ON on the remaining cases, and then uses ON's predictions to decide which suspicious cases to send for human relabelling, retraining O whenever a noisy label is found (Algorithm 1). The approach is evaluated on 552 Codeflaws subjects within the LEARN2FIX framework at 5%, 10%, and 20% injected label noise, reporting detection accuracy above 67% for a majority of subjects and a small number of relabelling queries. The authors also provide a public repository with implementation and data.","tokens_in":11248,"tokens_out":3174,"duration_ms":37265,"significance":"If the claims hold, ISONOISE would be a useful contribution: it addresses a real problem in HIOL, works with very small training suites (around 20-50 cases), and does not require an additional labelled dataset. The paper has concrete strengths: it builds on an existing method (Bouguelia et al.), evaluates on a substantial benchmark (552 subjects, 30 repetitions per subject), and ships reproducibility artifacts. However, the evaluation currently measures only recall-like accuracy, omits precision and false-alarm costs, compares against no noisy-label baseline, and leaves the dependence on the corrupted oracle's quality unquantified. The central idea is defensible, but the evidence as presented does not yet support the strength of the claimed practical benefit.","major_comments":[{"comment":"The 'detection accuracy' metric is effectively recall: the fraction of actually mislabelled test cases that ISONOISE flags. Precision and false-positive rate are never reported, so a reader cannot tell how many relabelling queries are wasted on correctly labelled cases. This is load-bearing because RQ.2's 'small number of relabelling queries' claim depends on both the number of queries and their utility; a method that sends every case for relabelling would achieve 100% recall but be useless. Please report a confusion-matrix breakdown (true positives, false positives, false negatives) or at least precision and false-alarm rate per noisy-label threshold, and define the accuracy metric explicitly in Section IV-C.","section":"Section V-A, Fig. 1"},{"comment":"The disagreement signal relies on labels assigned by the already-compromised oracle O to fuzzed mutants (t' labelled by O). When O is corrupted, the disagreement score measures agreement with a potentially wrong label, not with ground truth. The paper acknowledges this in Section V.A ('A highly compromised oracle adversely affects the disagreement calculation process in Algorithm 2'), but it never quantifies how ISONOISE's recall degrades as a function of O's accuracy. This is precisely the regime the technique is meant to address. Please add a controlled study that varies O's quality (e.g., by injecting different levels of noise before running ISONOISE, or by corrupting O directly) and plots detection recall and precision against O accuracy, so the operating envelope of Algorithm 2 is explicit.","section":"Section III-A, Algorithm 2, Line 6"},{"comment":"The experimental setup lists only 5%, 10%, and 20% noisy-label thresholds (Section IV-C: 'The above process is repeated for the 5%, 10% and 20% noisy label thresholds.'), and Fig. 2a/2b plot only these three thresholds. Yet Section V.B states 'In the 30% noisy label threshold, this probability exceeds 0.5' and later compares against a random baseline probability of 0.3 at 30%. A 30% condition appears neither in the setup nor in the figures, making the reported result unreproducible. Please either add the 30% threshold to the experiments and figures, or remove the unsubstantiated 30% references and any conclusions drawn from them.","section":"Section IV-C and Section V-B"},{"comment":"The disagreement threshold D=15 is the key control for separating 'highly disagreeing' from 'slightly disagreeing' test cases in Algorithm 1, yet it is set based only on unspecified pilot experiments and no sensitivity analysis is provided. The conclusions in Section V could depend critically on this single value. Please report how detection accuracy and relabelling queries vary as D changes (e.g., a sweep over a range around 15), and similarly for the fuzzing iteration count N=20, so the reader can judge the robustness of the reported results.","section":"Section IV-C, Disagreement Threshold, and Section V"},{"comment":"The paper claims that ISONOISE 'significantly increases the likelihood of sending an incorrectly labelled test for relabelling' (Introduction contribution 3 and Section V.B Result), but no statistical significance tests, confidence intervals, or effect-size estimates are provided; Fig. 1 and Fig. 2 are box plots without error bars or significance annotations. Given that each experiment is repeated 30 times, the data are sufficient to report standard errors or paired comparisons against a random relabelling baseline with the same query budget. Please add such statistical support, otherwise the word 'significantly' is unsupported.","section":"Section V, RQ.2"}],"minor_comments":[{"comment":"There is a typo 'incorreclty' that should read 'incorrectly'.","section":"Introduction, first paragraph of the contribution description"},{"comment":"Line 16 says 'Let hold be the previous human label of tn', but the loop variable is ts, not tn; this makes the pseudocode inconsistent with the surrounding text.","section":"Algorithm 1, Line 16"},{"comment":"The sentence 'Change the label of to hnew' is missing the object; it should read 'Change the label of ts to hnew'.","section":"Algorithm 1, Line 21"},{"comment":"The captions do not define 'Failing-Incorrect' and 'Passing-Incorrect', nor do they state whether the plotted values are medians or means across subjects; please add explicit definitions and clarify the aggregation procedure.","section":"Fig. 1 and Fig. 2 captions"},{"comment":"The text says 'The Disagreement Threshold was decided based on some pilot experiments' but gives no details about these pilots (number of subjects, range of values tried, criterion for selection). A sentence or two describing the pilot procedure would aid reproducibility.","section":"Section IV-C, 'Disagreement Threshold'"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper introduces ISONOISE, a technique to spot mislabelled test cases in human-in-the-loop oracle learning (HIOL) when the training suite is only 20–50 cases and there is no unlabelled pool. That setting is real, and as far as I can tell ISONOISE is the first method aimed at it. The core trick is to add fuzzed mutants labelled by the current oracle to the training set, train an intermediate classifier, and measure disagreement with the original human label. It borrows deliberately from Bougelia et al. and BDAL, and the iterative relabelling loop is a reasonable adaptation.\n\nWhat the paper does well: it runs on 552 Codeflaws subjects, reports detection accuracy and relabelling effort, and the code and data are on GitHub. It also openly admits that a compromised oracle degrades the disagreement calculation (Section V.A), which is the main conceptual risk. That honesty counts.\n\nThe soft spots are mostly in the measurement. 'Accuracy' here is recall-like; there is no precision or false-alarm rate, so we do not know how many relabelling queries are wasted. There is no comparison against a random-query baseline or an existing noisy-label method, and the disagreement threshold D=15 is tuned on pilot data with no sensitivity analysis. The box plots have no confidence intervals. Those are fixable, but they matter because the headline 'over 67%' is not the full picture.\n\nThe more serious issue is Algorithm 1. If no test case in TS meets the relabel condition, NoNoiseFound stays False and the while loop never terminates. And when one relabelled test case comes back with the same label, the loop stops and ignores the rest of TS. That is a termination bug, not a stylistic quibble, and it needs fixing before the method can be called a procedure. The 30% noisy threshold in Fig. 2b also contradicts the 5/10/20% setup.\n\nThe circular-oracle worry from the stress test is real but already disclosed; the paper should quantify how detection varies with oracle quality instead of just admitting the dependence. I would not call it a fatal flaw, because the external benchmark and golden labels break the circularity of the evaluation itself.\n\nBottom line: the paper deserves a serious referee and major revision. It is a niche but legitimate contribution to software testing, and the author is thinking clearly about the problem. For my own work I would probably not cite it in the next year, but I would bring it to a reading group focused on oracle learning.","headline":"ISONOISE is a plausible new noisy-label detector for tiny HIOL training sets; the idea is worth a serious referee, but the evaluation overstates precision and the main loop has a bug.","tokens_in":11809,"tokens_out":3046,"would_cite":false,"duration_ms":33506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ISONOISE isolates human mislabeled test cases with over 67% accuracy.","keywords":["noisy label detection","human-in-the-loop oracle learning","test oracle automation","active learning","software testing","label noise","automatic program repair"],"falsifier":"Take a Codeflaws subject, invert 30% of the labels randomly, and deliberately use an oracle O that has been trained only on the inverted labels so that its accuracy is near chance; if ISONOISE's detection accuracy drops to roughly 50% while it still sends many relabeling queries, the disagreement calculation is too dependent on the corrupted oracle to be reliable.","tokens_in":10796,"feed_emoji":"🧪","tokens_out":4230,"duration_ms":39847,"temperature":0.7,"pith_summary":"The paper claims that human labeling mistakes in human-in-the-loop oracle learning can be found automatically, without any extra labeled data. It introduces ISONOISE, which scores each test case by how often a family of classifiers trained on fuzzed variants disagrees with the human's label. Test cases with high disagreement are presented to the human for relabeling, and the oracle is retrained whenever a mistake is confirmed. On 552 buggy programs from Codeflaws, ISONOISE identifies the mislabeled test cases with over 67% accuracy for most subjects, while sending only a handful of relabeling queries. If true, this makes human-in-the-loop oracle learning and the program repair built on it more robust to noisy human labels.","feed_headline":"ISONOISE isolates human mislabeled test cases with over 67% accuracy","feed_subtitle":"A new method spots noisy labels in tiny human-in-the-loop training suites without extra labeled data.","key_machinery":"The load-bearing mechanism is the disagreement score from Algorithm 2, adapted from Bougelia et al.'s agree-to-disagree idea. For a test case $t$, the algorithm excludes $t$, fuzzes it $N$ times to produce mutants, labels each mutant with the existing oracle $O$, trains a new classifier on the suite plus the mutant, and increments disagreement whenever that classifier's prediction for $t$ differs from the human's label. This score separates suspicious test cases from trustworthy ones; then an intermediate classifier $O_N$ trained only on the low-disagreement cases decides which suspicious cases to send for relabeling. The threshold $D$ (set to 15 in the experiments) and the retraining loop are what make the method work with only 20--50 training examples.","core_discovery":"The central claim is that noisy labels introduced by the human during LEARN2FIX-style oracle learning can be isolated using only the already-trained automatic oracle and its training suite. ISONOISE computes a disagreement score for each test case by repeatedly mutating it through arithmetic fuzzing, labelling the mutant with the existing oracle, retraining a fresh classifier on the suite with the mutant added, and checking whether that classifier disagrees with the human's label. Test cases whose disagreement score exceeds a threshold are set aside, an intermediate oracle is trained on the remaining cases, and only the set-aside cases are candidates for relabeling. When a relabeling confirms a mistake, the oracle is retrained and the whole process restarts. The paper reports that for the majority of 552 Codeflaws subjects, the method detects the mislabeled cases with accuracy above 67%, with median accuracy above 70% for failing tests mislabeled as passing and above 60% for passing tests mislabeled as failing.","pith_inferences":["The disagreement-score approach might transfer to oracle learning for string inputs if fuzzing mutants are generated by grammar-based mutations, though the paper leaves this to future work.","The sensitivity of the disagreement threshold $D$ is likely the main practical tuning point; a data-driven way to set $D$ from the distribution of scores could make ISONOISE more robust across subjects.","Because detection accuracy falls as label noise increases, one could add an explicit early stop: if the intermediate oracle's confidence on relabeled cases is low, stop rather than risk confirming wrong labels.","The paper's 67% benchmark suggests a practical upper bound on how much label noise human-in-the-loop oracle learning can tolerate and still be corrected; estimating that bound per subject could guide when to invest in a reference implementation for relabeling."],"forward_implications":["If ISONOISE works as reported, human-in-the-loop oracle learning no longer has to assume perfect labeling; it can detect and correct a large share of human mistakes.","Since failing tests mislabeled as passing are found at median accuracy above 70%, the failure condition learned by LEARN2FIX is less likely to be silently corrupted.","The relabeling burden is small: the median number of relabeling queries is only slightly above the number of actual mislabeled tests, and the probability that a query hits a mislabeled test is far above random.","The same disagreement-driven isolation could be retrofitted to other oracle-learning and program-repair pipelines that suffer from label noise.","Accuracy degrades as the noise threshold rises, so the method is most useful when the human error rate is moderate (5--20%)."],"supporting_citations":[{"why":"Supplies the agree-to-disagree disagreement score that Algorithm 2 adapts.","marker":"[8]"},{"why":"Defines LEARN2FIX's human-in-the-loop oracle learning and documents how mislabeled tests degrade oracle and patch quality.","marker":"[5]"},{"why":"Provides the Codeflaws benchmark of 3,902 real-world faulty C programs from which the 552 subjects are selected.","marker":"[9]"},{"why":"Earlier LEARN2FIX description; the decision-tree classification used in the experiments comes from this line of work.","marker":"[6]"},{"why":"Arithmetic mutation operators used by MUTATE FUZZ to generate the fuzzed test cases for disagreement calculation.","marker":"[24]"},{"why":"Bi-directional active learning, whose incremental oracle rectification inspired ISONOISE's retraining loop.","marker":"[12]"}],"fun_headline_variants":["ISONOISE spots mislabelled test cases in human-in-the-loop oracle learning","Over 67% accuracy: ISONOISE finds human label mistakes in oracle training","Trace noisy labels in oracle learning: ISONOISE hits >67% accuracy","Isolate bad test labels in oracle learning: ISONOISE at 67%+ accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The disagreement scores keep working even when the oracle used to label the fuzzed mutants is itself partly corrupted by the very mislabeled tests we are trying to find.","fun_headline_variants_meta":{"raw":{"variants":["ISONOISE spots mislabelled test cases in human-in-the-loop oracle learning","Over 67% accuracy: ISONOISE finds human label mistakes in oracle training","Trace noisy labels in oracle learning: ISONOISE hits >67% accuracy","Isolate bad test labels in oracle learning: ISONOISE at 67%+ accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4438,"prompt_tokens":993,"completion_tokens":3445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3354}},"tokens_in":609,"tokens_out":3445,"duration_ms":26351,"temperature":1.0,"reasoning_tokens":3354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:34.542907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a Codeflaws subject, invert 30% of the labels randomly, and deliberately use an oracle O that has been trained only on the inverted labels so that its accuracy is near chance; if ISONOISE's detection accuracy drops to roughly 50% while it still sends many relabeling queries, the disagreement calculation is too dependent on the corrupted oracle to be reliable.","supporting_citations":[{"cited_title":"Agreeing to disagree: Active learning with noisy labels without crowdsourcing,","cited_arxiv_id":null,"evidence_quote":"Supplies the agree-to-disagree disagreement score that Algorithm 2 adapts."},{"cited_title":"Human-in-the-loop automatic program repair,","cited_arxiv_id":null,"evidence_quote":"Defines LEARN2FIX's human-in-the-loop oracle learning and documents how mislabeled tests degrade oracle and patch quality."},{"cited_title":"Codeflaws: A programming competition benchmark for evaluating automated pro- gram repair tools,","cited_arxiv_id":null,"evidence_quote":"Provides the Codeflaws benchmark of 3,902 real-world faulty C programs from which the 552 subjects are selected."},{"cited_title":"Human-in-the-loop automatic program repair,","cited_arxiv_id":null,"evidence_quote":"Earlier LEARN2FIX description; the decision-tree classification used in the experiments comes from this line of work."},{"cited_title":"Bidirectional active learning: A two-way exploration into unlabeled and labeled data set,","cited_arxiv_id":null,"evidence_quote":"Bi-directional active learning, whose incremental oracle rectification inspired ISONOISE's retraining loop."}],"review_version":1}