{"id":"60ac73e5-c1f8-434f-a05a-d644b1b1c7f4","arxiv_id":"2412.12340","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuned Mistral-7b, Llama2-7b, and CodeLlama-7b classify flaky test root causes in C++ and Java, with Mistral-7b reporting perfect scores on a small augmented C++ dataset.","lead":"This paper fine-tunes three 7-billion-parameter language models to label the root cause of flaky tests in C++ projects, and reports that one model reaches a perfect score on its new dataset. The result matters because flaky tests are a costly problem in continuous integration, and an automated classifier could save debugging effort.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Section 5 concedes that random train-test splitting can put GPT-4-mutated copies of a training original into the C++ evaluation set; the reported perfect Mistral-7b score is therefore not evidence of generalization. A family-level regrouped split is the decisive check.","rationale":"The load-bearing claim is the perfect C++ score; if that falls, the main contribution collapses to a dataset artifact paper. I agree with the reader's weakest_assumption: the acknowledged leakage is the most direct threat. I would not center a complaint about small dataset size or category imbalance alone, because a perfect score on about 90 examples is surprising even with imbalance; the family-similarity mechanism explains how it could happen and is admitted in Section 5. The Java comparison is also defective (Table 6 mixes C++ and Java rows), but that is secondary because the abstract's 'exceptional capability' claim rests mainly on C++ 1.0. The paper does provide a new public dataset and reproducible scripts, which are genuine contributions, but the experimental claim of generalization is not supported as reported. The reader's REJECT verdict is therefore unchanged by this stress-test; the paper would need a family-level split and corrected comparisons before its central claim could be accepted.","tokens_in":14923,"tokens_out":3833,"duration_ms":33139,"concrete_test":"Use the public GitHub dataset to reproduce the C++ experiment with a family-level split: group each original GitHub issue with all of its GPT-4-generated variants into one unit, then randomly assign units to 75% training / 25% evaluation. Fine-tune Mistral-7b with the same hyperparameters and report weighted F1, precision, recall, and test-set size. If the C++ F1 drops materially below 1.0 (say below 0.90) or per-class recall collapses for small classes, the perfect score is an artifact of split leakage. Also report, for the original split, the fraction of evaluation instances whose source original appears in training; if it is high, leakage is directly quantified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the perfect C++ classification (Mistral-7b, F1 = 1.0, Table 4). That claim requires the evaluation set to be independent of the training set. The C++ dataset is constructed from 55 manually collected tests plus GPT-4-generated variants: each original is mutated by renaming variables/constants, renaming test methods, and adding unused variable declarations (Section 3.1, Figure 2). The final dataset has 362 instances, so the random 75/25 split yields only about 90 evaluation instances, and each original's mutated siblings are highly similar to the original. If an original lands in training and its variants land in testing, the model can label a test by recognizing the family rather than by learning flakiness categories. The authors explicitly acknowledge this in Section 5: 'due to our random train-test splitting strategy, there is a possibility that the original tests in the training set influenced the classification of corresponding mutated tests in the evaluation set. This could have inflated the model's performance, as LLMs like GPT-4 have been shown to recognize mutations.' This is not an external objection; it is a stated threat to the reported result. Because the perfect C++ score is the headline result and the basis for the 'exceptional capability' conclusion, the leakage concern is load-bearing. A secondary issue is that Table 6 compares FlakyCat's Java results against C++ rows for the proposed models, so RQ2 is not cleanly supported either.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning three 7-billion-parameter LLMs (Mistral-7b, Llama2-7b, CodeLlama-7b) with LoRA to classify the root cause of flaky tests in C++ and Java code. The authors construct a C++ flaky-test dataset of 55 manually collected tests, augmented with GPT-4-generated variants to 362 instances, and evaluate on this dataset plus the existing FlakyCat Java dataset (1287 instances). They report that Mistral-7b achieves a perfect 1.0 F1/precision/recall/accuracy on the C++ dataset, that Llama2-7b performs best on Java (F1 0.89), and that all three models outperform FlakyCat's CodeBERT-based classifier on Java. The paper claims exceptional capability of LLMs for flakiness classification and releases the dataset and scripts on GitHub.","tokens_in":15235,"tokens_out":5540,"duration_ms":44213,"significance":"If the reported results were credible, the paper would provide a valuable first C++ flaky-test dataset and evidence that a small fine-tuned LLM can classify flakiness root causes from source code alone. The public dataset and scripts are a useful reproducibility contribution. However, the central C++ result is undermined by the data-leakage problem the authors themselves acknowledge, and the RQ2 comparison is misreported. The significance of the empirical claims therefore depends on whether a corrected evaluation, e.g., with family-level splitting, still shows high performance.","major_comments":[{"comment":"The reported perfect C++ classification (Table 4, Mistral-7b F1=1.0) is not credible evidence of generalization because the random 75/25 split can place an original collected test in training and its GPT-4-generated mutated copies in evaluation. The mutations described in Section 3.1 (renaming variables/constants, renaming test method names, adding unused variable declarations) are superficial, so the model could memorize the original test's family rather than learn a general mapping from code to flakiness category. The authors explicitly acknowledge this in Section 5: 'there is a possibility that the original tests in the training set influenced the classification of corresponding mutated tests in the evaluation set.' The decisive check is a family-level split, where all mutations derived from the same original test are kept in the same partition; the paper currently does not provide such an evaluation.","section":"Section 5, Section 3.1"},{"comment":"The RQ2 comparison is internally inconsistent. Table 6 lists 'Our Study' rows with F1/precision/recall of 1.0, 0.90, 0.79 for Mistral-7b, Llama2-7b, CodeLlama-7b, which are exactly the C++ values from Table 4, but the surrounding text and RQ2 concern the Java dataset. The text states 'Llama2-7b model achieved the highest F1 score of 0.89,' which matches the Java row of Table 4, not the 0.90 shown in Table 6. The table must be corrected to report the Java results (0.85, 0.89, 0.86) when comparing against FlakyCat, or the comparison should be clearly labeled as a cross-dataset comparison.","section":"Section 4, Table 6"},{"comment":"The per-category results include categories that are not part of the datasets described in Section 3.1. The C++ rows contain a 'Hash operation' category, but Table 1 and Table 3 do not define or list this category for C++; the C++ dataset contains only Async wait, Concurrency, Time, Unordered collections, Float point operation, I/O, Randomness, and Too restrictive range. Similarly, the Java rows contain a 'Test case timeout' category, while Table 3's Java dataset does not include this category. Either the dataset description is incomplete or the table lists results for an undisclosed evaluation setup. This must be reconciled for the results to be reproducible.","section":"Section 4, Table 5"}],"minor_comments":[{"comment":"The prompt instructs GPT-4 to 'use the SMOTE method,' but SMOTE is an oversampling technique for numeric feature vectors, not a code mutation procedure; the description should clarify that SMOTE is only an inspiration and the actual augmentation is performed by GPT-4.","section":"Section 3.1"},{"comment":"Figure numbering is inconsistent: the text refers to 'Fig 2 and Fig 3' for the accuracy/F1 curves, but Figures 3 and 4 are the actual result figures, while Figure 2 is a code snippet.","section":"Section 4"},{"comment":"Table 4 does not state whether the reported metrics are from the final epoch or the best epoch across the ten training epochs; this should be specified for reproducibility.","section":"Section 4, Table 4"},{"comment":"Section 7 refers to 'Llama-7b' while the rest of the paper uses 'Llama2-7b'; unify the model name.","section":"Section 7"},{"comment":"The abstract and Section 4 say 'achieving a score of 1' and 'a score of 1.0'; use a consistent numerical format (1.0) throughout.","section":"Abstract, Section 4"},{"comment":"In Table 5, the category 'Float point operation' is labeled differently from 'Floating Point Operations' in Table 1; unify the naming for clarity.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent about the data-leakage risk, which is commendable, but the current evaluation does not support the headline claim of perfect C++ classification. The paper is more suitable as a preliminary dataset-and-baseline paper; if the evaluation is redone with a family-level split, the results may change substantially. Please consider whether the venue's standards require the corrected evaluation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the new C++ dataset is a real contribution, and the paper is honest about its main weakness, but the headline result—Mistral-7b with 1.0 F1 on C++—does not hold up as evidence of generalization. The authors themselves concede in Section 5 that random splitting can put GPT-4-mutated copies of a training original into the evaluation set. With 55 originals and 307 augmented variants, the test set of ~90 instances is likely dominated by family members of training tests, so the model can recognize mutations rather than classify flakiness. That is the load-bearing claim, and it fails.\n\nWhat is genuinely new: this is the first publicly available C++ flaky test dataset of which I'm aware, with 362 instances and labels following the standard taxonomy. The authors also fine-tuned three 7B LLMs with LoRA and report per-category results. They make code, data, and scripts public, and they flag the leakage threat themselves—credit where due.\n\nThe soft spots beyond leakage: Table 6 compares FlakyCat's Java numbers against the proposed models' C++ numbers, so RQ2 (Java comparison) is not cleanly supported. The Java and C++ datasets share only four categories, which makes RQ1's comparison weaker than it looks. The dataset is small and heavily augmented, and the manual labeling of 55 tests is a construct validity concern. None of these are fatal in themselves, but combined they mean the paper's current conclusions outrun the evidence.\n\nWho this paper is for: anyone working on flakiness classification, especially in C++, and anyone who wants a benchmark to test LLM-based classifiers. The dataset artifact is worth citing with caution.\n\nMy recommendation: I would not desk-reject this. The dataset is valuable, the authors are transparent, and the fix is not hard—regroup by test family before splitting, or evaluate only on genuinely held-out originals. A serious referee should ask for that redesign and a corrected Table 6 before any claims about exceptional capability are made. As it stands, the paper is a promising dataset paper in need of an honest evaluation.","headline":"Useful C++ flaky-test dataset, but the perfect Mistral score is a leakage artifact and Table 6 muddles the Java comparison; worth a serious referee only if the split is fixed.","tokens_in":15769,"tokens_out":2475,"would_cite":true,"duration_ms":24003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned Mistral-7b model classifies the root cause of C++ flaky tests from code alone, scoring 1.0 across all metrics on the authors' dataset and beating a CodeBERT-based baseline on Java.","keywords":["flaky tests","root cause classification","large language models","fine-tuning","C++ testing","test flakiness","data augmentation","continuous integration"],"falsifier":"Re-run the C++ evaluation with a grouped split that keeps each original test and all of its mutated variants in the same subset, or train on the 55 original tests and test on the 307 augmented ones; if Mistral-7b's perfect scores fall sharply, the original result came from data leakage rather than learned generalization.","tokens_in":14720,"feed_emoji":"🧪","tokens_out":8939,"duration_ms":70733,"temperature":0.7,"pith_summary":"This paper tries to establish that a fine-tuned 7-billion-parameter large language model can classify the root cause of a flaky test from the test code alone, for C++ as well as Java. The authors built a C++ flaky-test dataset of 362 cases, starting from 55 real-world tests and expanding them with GPT-4-generated variants designed to preserve the underlying flakiness. They report that Mistral-7b achieves a perfect score across all metrics on this C++ dataset, and that all three fine-tuned models outperform an existing CodeBERT-based classifier on the Java benchmark. If the claim holds, automated root-cause classification could let developers skip repeated test re-runs and go straight to the part of the test that needs fixing.","feed_headline":"Fine-tuned Mistral-7b classifies C++ flaky tests perfectly","feed_subtitle":"Tuned 7B models beat the CodeBERT baseline on Java and hit F1 1.0 on the new C++ flaky-test dataset.","key_machinery":"The machinery is a fine-tuning pipeline built around 7-billion-parameter decoder-only language models. Each pretrained model is adapted with LoRA, which freezes the original weights and inserts small trainable low-rank matrices, then trained on test code paired with root-cause labels such as Async Wait, Concurrency, Time, Randomness, and Unordered Collections. The C++ training set is expanded by prompting GPT-4 to rename variables and constants and to add unused declarations without changing the test's flakiness, following a SMOTE-inspired oversampling idea. The load-bearing mechanism is the assumption that those mutated copies carry the same root cause as the originals, so a model that learns the category from one version should recognize it in another.","core_discovery":"The central claim is that flakiness root-cause classification works from source code alone and transfers to C++. The authors fine-tune Mistral-7b, Llama2-7b, and CodeLlama-7b with LoRA on their new C++ dataset and on the Java dataset of a prior CodeBERT-based study. According to the reported results, Mistral-7b classifies the C++ evaluation set perfectly, with accuracy, precision, recall, and F1 all equal to 1.0; Llama2-7b reaches 0.90 on C++ and the best Java F1 at 0.89; and every fine-tuned model beats the prior classifier's 0.73 F1 on Java. The paper also claims to contribute the first public C++ flaky-test dataset, labeled using an established root-cause taxonomy.","pith_inferences":["If the augmentation-leakage risk materializes, the reported C++ models may be ranked incorrectly; forcing each original test and its mutated copies into the same split is a direct way to test this.","Because the C++ labels are derived from developer comments plus the authors' manual review, the classifier's ceiling is the consistency of that subjective labeling, not an objective ground truth.","The finding that a general-purpose LLM beat a code-specialized one, if it survives a leakage-free split, suggests flakiness classification rewards semantic understanding of tests more than code-generation skill.","The same original-plus-mutation split hazard applies to any benchmark built by augmenting a small seed set, so future flaky-test datasets should report both random and grouped split results."],"forward_implications":["A fine-tuned 7B model could be inserted into a continuous-integration pipeline to label newly failing tests by root cause without rerunning them many times.","The Java results indicate that fine-tuned decoder LLMs surpass the CodeBERT-based approach, so future flakiness classifiers should consider this model family.","Per-language model choice matters: the authors recommend Mistral-7b for C++ and Llama2-7b for Java based on measured F1 scores.","CodeLlama-7b underperforming Llama2-7b shows that code-specialized pretraining does not automatically improve classification tasks such as this one.","LoRA fine-tuning completed in under six minutes on the used hardware, which makes per-project or per-language fine-tuning practical in industry settings."],"supporting_citations":[{"why":"Supplies the Java dataset, the root-cause labels, and the CodeBERT-based few-shot classifier whose 0.73 F1 serves as the baseline comparison.","marker":"[7]"},{"why":"Supplies the ten-category root-cause taxonomy used to label the flaky tests in this study.","marker":"[20]"},{"why":"Defines the Mistral-7b architecture that becomes the best-performing C++ classifier.","marker":"[38]"},{"why":"Provides Llama2-7b, the base model for two of the three fine-tuned classifiers and the best Java performer.","marker":"[37]"},{"why":"GPT-4 is used to generate the augmented C++ test variants via the prompted mutation procedure.","marker":"[31]"},{"why":"SMOTE is the oversampling method named in the augmentation prompt and the reference for the data-expansion strategy.","marker":"[32]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train all models.","marker":"[39]"}],"fun_headline_variants":["Mistral-7b perfect F1 on C++ flaky tests","LLM fine-tuning identifies C++ flaky tests perfectly","Fine-tuned Mistral beats baseline, perfect on C++ flakiness","First C++ flaky-test dataset, Mistral-7b classifies with F1 1.0"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random 75/25 train-test split creates an independent evaluation set; the paper itself notes in its discussion that original tests in the training set may have influenced the classification of their GPT-4-generated mutated copies in the evaluation set, which would inflate the reported C++ scores if the model recognized the mutations.","fun_headline_variants_meta":{"raw":{"variants":["Mistral-7b perfect F1 on C++ flaky tests","LLM fine-tuning identifies C++ flaky tests perfectly","Fine-tuned Mistral beats baseline, perfect on C++ flakiness","First C++ flaky-test dataset, Mistral-7b classifies with F1 1.0"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1807,"prompt_tokens":1001,"completion_tokens":806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":729}},"tokens_in":617,"tokens_out":806,"duration_ms":6884,"temperature":1.0,"reasoning_tokens":729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:10:20.108826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the C++ evaluation with a grouped split that keeps each original test and all of its mutated variants in the same subset, or train on the 55 original tests and test on the 307 augmented ones; if Mistral-7b's perfect scores fall sharply, the original result came from data leakage rather than learned generalization.","supporting_citations":[{"cited_title":"Flakycat: predicting flaky tests categories using few-shot learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the Java dataset, the root-cause labels, and the CodeBERT-based few-shot classifier whose 0.73 F1 serves as the baseline comparison."},{"cited_title":"An empirical analysis of flaky tests,","cited_arxiv_id":null,"evidence_quote":"Supplies the ten-category root-cause taxonomy used to label the flaky tests in this study."},{"cited_title":"Smote: synthetic minority oversampling technique,","cited_arxiv_id":null,"evidence_quote":"SMOTE is the oversampling method named in the augmentation prompt and the reference for the data-expansion strategy."}],"review_version":1}