Pith. sign in

REVIEW 3 major objections 6 minor 43 references

A Large Language Model Approach to Identify Flakiness in C++ Projects

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned Mistral-7b model classifies the root cause of C++ flaky tests from code alone, scoring 1.0 across all metrics on the authors' dataset and beating a CodeBERT-based baseline on Java.

desk verdict Useful C++ flaky-test dataset, but the perfect Mistral score is a leakage artifact and Table 6 muddles the Java comparison; worth a serious referee only if the split is fixed. read the letter →

arxiv 2412.12340 v2 pith:KWCMZ6IQ submitted 2024-12-16 cs.SE

Xin Sun , Daniel St{aa}hl , Kristian Sandahl This is my paper · ORCID
classification cs.SE
keywords flakytestsrootcauseclassificationlargelanguagemodelsfine-tuningC++testingtestflakinessdataaugmentationcontinuousintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a fine-tuned 7-billion-parameter large language model can classify the root cause of a flaky test from the test code alone, for C++ as well as Java. The authors built a C++ flaky-test dataset of 362 cases, starting from 55 real-world tests and expanding them with GPT-4-generated variants designed to preserve the underlying flakiness. They report that Mistral-7b achieves a perfect score across all metrics on this C++ dataset, and that all three fine-tuned models outperform an existing CodeBERT-based classifier on the Java benchmark. If the claim holds, automated root-cause classification could let developers skip repeated test re-runs and go straight to the part of the test that needs fixing.

What carries the argument

The machinery is a fine-tuning pipeline built around 7-billion-parameter decoder-only language models. Each pretrained model is adapted with LoRA, which freezes the original weights and inserts small trainable low-rank matrices, then trained on test code paired with root-cause labels such as Async Wait, Concurrency, Time, Randomness, and Unordered Collections. The C++ training set is expanded by prompting GPT-4 to rename variables and constants and to add unused declarations without changing the test's flakiness, following a SMOTE-inspired oversampling idea. The load-bearing mechanism is the assumption that those mutated copies carry the same root cause as the originals, so a model that learns the category from one version should recognize it in another.

What would settle it

Re-run the C++ evaluation with a grouped split that keeps each original test and all of its mutated variants in the same subset, or train on the 55 original tests and test on the 307 augmented ones; if Mistral-7b's perfect scores fall sharply, the original result came from data leakage rather than learned generalization.

Watch

Extended reading notes

Core claim

The central claim is that flakiness root-cause classification works from source code alone and transfers to C++. The authors fine-tune Mistral-7b, Llama2-7b, and CodeLlama-7b with LoRA on their new C++ dataset and on the Java dataset of a prior CodeBERT-based study. According to the reported results, Mistral-7b classifies the C++ evaluation set perfectly, with accuracy, precision, recall, and F1 all equal to 1.0; Llama2-7b reaches 0.90 on C++ and the best Java F1 at 0.89; and every fine-tuned model beats the prior classifier's 0.73 F1 on Java. The paper also claims to contribute the first public C++ flaky-test dataset, labeled using an established root-cause taxonomy.

Load-bearing premise

The load-bearing premise is that the random 75/25 train-test split creates an independent evaluation set; the paper itself notes in its discussion that original tests in the training set may have influenced the classification of their GPT-4-generated mutated copies in the evaluation set, which would inflate the reported C++ scores if the model recognized the mutations.

Editorial extensions

If this is right

  • A fine-tuned 7B model could be inserted into a continuous-integration pipeline to label newly failing tests by root cause without rerunning them many times.
  • The Java results indicate that fine-tuned decoder LLMs surpass the CodeBERT-based approach, so future flakiness classifiers should consider this model family.
  • Per-language model choice matters: the authors recommend Mistral-7b for C++ and Llama2-7b for Java based on measured F1 scores.
  • CodeLlama-7b underperforming Llama2-7b shows that code-specialized pretraining does not automatically improve classification tasks such as this one.
  • LoRA fine-tuning completed in under six minutes on the used hardware, which makes per-project or per-language fine-tuning practical in industry settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the augmentation-leakage risk materializes, the reported C++ models may be ranked incorrectly; forcing each original test and its mutated copies into the same split is a direct way to test this.
  • Because the C++ labels are derived from developer comments plus the authors' manual review, the classifier's ceiling is the consistency of that subjective labeling, not an objective ground truth.
  • The finding that a general-purpose LLM beat a code-specialized one, if it survives a leakage-free split, suggests flakiness classification rewards semantic understanding of tests more than code-generation skill.
  • The same original-plus-mutation split hazard applies to any benchmark built by augmenting a small seed set, so future flaky-test datasets should report both random and grouped split results.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes fine-tuning three 7-billion-parameter LLMs (Mistral-7b, Llama2-7b, CodeLlama-7b) with LoRA to classify the root cause of flaky tests in C++ and Java code. The authors construct a C++ flaky-test dataset of 55 manually collected tests, augmented with GPT-4-generated variants to 362 instances, and evaluate on this dataset plus the existing FlakyCat Java dataset (1287 instances). They report that Mistral-7b achieves a perfect 1.0 F1/precision/recall/accuracy on the C++ dataset, that Llama2-7b performs best on Java (F1 0.89), and that all three models outperform FlakyCat's CodeBERT-based classifier on Java. The paper claims exceptional capability of LLMs for flakiness classification and releases the dataset and scripts on GitHub.

Significance. If the reported results were credible, the paper would provide a valuable first C++ flaky-test dataset and evidence that a small fine-tuned LLM can classify flakiness root causes from source code alone. The public dataset and scripts are a useful reproducibility contribution. However, the central C++ result is undermined by the data-leakage problem the authors themselves acknowledge, and the RQ2 comparison is misreported. The significance of the empirical claims therefore depends on whether a corrected evaluation, e.g., with family-level splitting, still shows high performance.

major comments (3)
  1. [Section 5, Section 3.1] The reported perfect C++ classification (Table 4, Mistral-7b F1=1.0) is not credible evidence of generalization because the random 75/25 split can place an original collected test in training and its GPT-4-generated mutated copies in evaluation. The mutations described in Section 3.1 (renaming variables/constants, renaming test method names, adding unused variable declarations) are superficial, so the model could memorize the original test's family rather than learn a general mapping from code to flakiness category. The authors explicitly acknowledge this in Section 5: 'there is a possibility that the original tests in the training set influenced the classification of corresponding mutated tests in the evaluation set.' The decisive check is a family-level split, where all mutations derived from the same original test are kept in the same partition; the paper currently does not provide such an evaluation.
  2. [Section 4, Table 6] The RQ2 comparison is internally inconsistent. Table 6 lists 'Our Study' rows with F1/precision/recall of 1.0, 0.90, 0.79 for Mistral-7b, Llama2-7b, CodeLlama-7b, which are exactly the C++ values from Table 4, but the surrounding text and RQ2 concern the Java dataset. The text states 'Llama2-7b model achieved the highest F1 score of 0.89,' which matches the Java row of Table 4, not the 0.90 shown in Table 6. The table must be corrected to report the Java results (0.85, 0.89, 0.86) when comparing against FlakyCat, or the comparison should be clearly labeled as a cross-dataset comparison.
  3. [Section 4, Table 5] The per-category results include categories that are not part of the datasets described in Section 3.1. The C++ rows contain a 'Hash operation' category, but Table 1 and Table 3 do not define or list this category for C++; the C++ dataset contains only Async wait, Concurrency, Time, Unordered collections, Float point operation, I/O, Randomness, and Too restrictive range. Similarly, the Java rows contain a 'Test case timeout' category, while Table 3's Java dataset does not include this category. Either the dataset description is incomplete or the table lists results for an undisclosed evaluation setup. This must be reconciled for the results to be reproducible.
minor comments (6)
  1. [Section 3.1] The prompt instructs GPT-4 to 'use the SMOTE method,' but SMOTE is an oversampling technique for numeric feature vectors, not a code mutation procedure; the description should clarify that SMOTE is only an inspiration and the actual augmentation is performed by GPT-4.
  2. [Section 4] Figure numbering is inconsistent: the text refers to 'Fig 2 and Fig 3' for the accuracy/F1 curves, but Figures 3 and 4 are the actual result figures, while Figure 2 is a code snippet.
  3. [Section 4, Table 4] Table 4 does not state whether the reported metrics are from the final epoch or the best epoch across the ten training epochs; this should be specified for reproducibility.
  4. [Section 7] Section 7 refers to 'Llama-7b' while the rest of the paper uses 'Llama2-7b'; unify the model name.
  5. [Abstract, Section 4] The abstract and Section 4 say 'achieving a score of 1' and 'a score of 1.0'; use a consistent numerical format (1.0) throughout.
  6. [Table 5] In Table 5, the category 'Float point operation' is labeled differently from 'Floating Point Operations' in Table 1; unify the naming for clarity.

Circularity Check

1 steps flagged · score 6.0 of 10

C++ perfect-classification claim is partially circular: the random split lets GPT-4-mutated copies of training tests enter the evaluation set, so the F1=1.0 result does not measure generalization.

  1. fitted input called prediction [Section 3.1 (Data Augmentation) and Section 3.2 (Fine-Tuning, 75/25 split), reported in Table 4; limitation acknowledged in Section 5]
    "due to our random train-test splitting strategy, there is a possibility that the original tests in the training set influenced the classification of corresponding mutated tests in the evaluation set. This could have inflated the model's performance, as LLMs like GPT-4 have been shown to recognize mutations."

    The C++ dataset is produced by prompting GPT-4 to mutate each of 55 collected tests into variants ('For each given code, please return five augmented examples to me'), then mixing originals with mutations into 362 samples. The 75/25 split is applied to that combined set, so there is no guarantee that a test and its mutated siblings stay in the same partition. If an original is in training and a GPT-4 mutation is in evaluation, the model has already seen the underlying test or a near copy, and can label the evaluation item by lexical/syntactic family matching rather than by a learned rule about flakiness categories. The paper acknowledges this exact mechanism and its performance-inflating consequence.

full rationale

The only load-bearing circularity is in the C++ evaluation. The paper builds its 362-sample C++ dataset by GPT-4-mutating 55 originals (Section 3.1), then randomly splits the combined set 75/25 (Section 3.2). Since mutated copies of an original can fall into the test set while the original or sibling mutations sit in training, the test set is not label-independent; a model can score 1.0 by recognizing mutation families rather than by classifying flakiness causes. The paper itself concedes this in Section 5, yet reports the perfect C++ score as the headline result. The Java comparison is not circular: it uses FlakyCat's dataset and an external CodeBERT baseline, though Table 6 confusingly lists C++ rows under 'Our Study' for the Java RQ2, which is a reporting error rather than a circularity. No self-citation chains or imported uniqueness theorems are load-bearing here.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several assumptions about data quality and split independence, the most fragile of which is the independence of the augmented test set.

free parameters (7)
  • LoRA rank = 12
    Set following Li et al. [40]; controls the rank of low-rank adaptation matrices.
  • LoRA alpha = 32
    Set following Li et al. [40]; scaling factor for LoRA updates.
  • LoRA dropout = 0.1
    Set following Li et al. [40]; dropout rate to reduce overfitting.
  • Learning rate = 5e-5
    Chosen by the authors; described as accelerating convergence.
  • Number of epochs = 10
    Chosen by the authors; the model passes through the dataset ten times.
  • Batch size = 8
    Chosen by the authors for training and evaluation.
  • Max sequence length = 512
    Chosen because most test cases are short.
assumptions (4)
  • domain assumption Manual labels assigned from developer comments are correct
    The dataset was labeled by the authors from GitHub issue comments and test code review; mislabeled examples would bias training and evaluation (Section 3.1, Section 6.1).
  • domain assumption GPT-4 data augmentation preserves the flakiness root cause
    The authors assume renaming variables and adding unused declarations does not change the flakiness category; the augmented tests were checked manually but the premise is not formally verified.
  • domain assumption Random 75/25 split yields independent training and evaluation sets
    The evaluation validity depends on this, but the paper's Section 5 explicitly doubts it because original tests and their mutations may be split across sets.
  • domain assumption The FlakyCat Java dataset is correctly labeled and comparable
    The paper uses 1287 Java tests from FlakyCat without re-validating labels or confirming the exact split used for the baseline comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Large Language Model Approach to Identify Flakiness in C++ Projects." pith.science (2026). https://pith.science/paper/KWCMZ6IQ

@misc{pith2026241212340,
  author       = {Pith},
  title        = {Pith review of: A Large Language Model Approach to Identify Flakiness in C++ Projects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWCMZ6IQ}},
  note         = {Machine review of arXiv:2412.12340}
}
read the original abstract

The role of regression testing in software testing is crucial as it ensures that any new modifications do not disrupt the existing functionality and behaviour of the software system. The desired outcome is for regression tests to yield identical results without any modifications made to the system being tested. In practice, however, the presence of Flaky Tests introduces non-deterministic behaviour and undermines the reliability of regression testing results. In this paper, we propose an LLM-based approach for identifying the root cause of flaky tests in C++ projects at the code level, with the intention of assisting developers in debugging and resolving them more efficiently. We compile a comprehensive collection of C++ project flaky tests sourced from GitHub repositories. We fine-tune Mistral-7b, Llama2-7b and CodeLlama-7b models on the C++ dataset and an existing Java dataset and evaluate the performance in terms of precision, recall, accuracy, and F1 score. We assess the performance of the models across various datasets and offer recommendations for both research and industry applications. The results indicate that our models exhibit varying performance on the C++ dataset, while their performance is comparable to that of the Java dataset. The Mistral-7b surpasses the other two models regarding all metrics, achieving a score of 1. Our results demonstrate the exceptional capability of LLMs to accurately classify flakiness in C++ and Java projects, providing a promising approach to enhance the efficiency of debugging flaky tests in practice.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 32 canonical work pages

  1. [1]

    Taming timeout flakiness: An empirical study of sap hana,

    A. Berndt, S. Baltes, and T. Bach, “Taming timeout flakiness: An empirical study of sap hana,” 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 69–80, 2024

    work page 2024

  2. [2]

    Software testing research challenges: An industrial perspective,

    N. Alshahwan, M. Harman, and A. Marginean, “Software testing research challenges: An industrial perspective,” in 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 1– 10, 2023

    work page 2023

  3. [3]

    Deflaker: Automatically detecting flaky tests,

    J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Mari nov, “Deflaker: Automatically detecting flaky tests,” in Proceedings of the 40th international conference on software engineering, pp. 433–444, 2018

    work page 2018

  4. [4]

    De-flake your tests: Automatically locating root causes of flaky tests in code at google,

    C. Ziftci and D. Cavalcanti, “De-flake your tests: Automatically locating root causes of flaky tests in code at google,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 736–745, IEEE, 2020

    work page 2020

  5. [5]

    What do developer -repaired flaky tests tell us about the effective ness of automated flaky test detection?,

    O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “What do developer -repaired flaky tests tell us about the effective ness of automated flaky test detection?,” in Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, pp. 160–164, 2022

    work page 2022

  6. [6]

    A multi -factor approach for flaky test detection and automated root cause analysis,

    A. Ahmad, F. G. de Oliveira Neto, Z. Shi, K. Sandahl, and O. Leifler, “A multi -factor approach for flaky test detection and automated root cause analysis,” in 2021 28th Asia -Pacific Software Engineering Conference (APSEC), pp. 338–348, IEEE, 2021

    work page 2021

  7. [7]

    Flakycat: predicting flaky tests categories using few-shot learning,

    A. Akli, G. Haben, S. Habchi, M. Papadakis, and Y. Le Traon, “Flakycat: predicting flaky tests categories using few-shot learning,” in 2023 IEEE/ACM International Conference on Automation of Software Test (AST), pp. 140–151, IEEE, 2023

    work page 2023

  8. [8]

    Continuous practices and devops: beyond the buzz, what does it all mean?,

    D. Stahl, T. Martensson, and J. Bosch, “Continuous practices and devops: beyond the buzz, what does it all mean?,” in 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 440–448, 2017

    work page 2017

Show all 43 references
  1. [9]

    A qualitative study on the sources, impacts, and mitigation strategies of fla ky tests,

    S. Habchi, G. Haben, M. Papadakis, M. Cordy, and Y. Le Traon, “A qualitative study on the sources, impacts, and mitigation strategies of fla ky tests,” in 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 244–255, IEEE, 2022

    2022

  2. [10]

    Understanding flaky tests: The developer’s perspective,

    M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: The developer’s perspective,” in Proceedings of th e 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 830–840, 2019

    2019

  3. [11]

    Taming google-scale continuous testing,

    A. M. Memon, Z. Gao, B.-N. Nguyen, S. Dhanda, E. Nickell, R. Siemborski, and J. Micco, “Taming google-scale continuous testing,” 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 233–242, 2017

    2017

  4. [12]

    Root causing flaky tests in a large-scale industrial setting,

    W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thummalapenta, “Root causing flaky tests in a large-scale industrial setting,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 20 19, (New York, NY, USA), p. 101 –111, Ass...

    2019

  5. [13]

    A survey of flaky tests,

    O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,” ACM Trans. Softw. Eng. Methodol., vol. 31, Oct. 2021

    2021

  6. [14]

    A survey on how test flakiness affects developers and what support they need to address it,

    M. Gruber and G. Fraser, “A survey on how test flakiness affects developers and what support they need to address it,” in 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 82–92, 2022. 218 Computer Science & Information Technology (CS & IT)

    2022

  7. [15]

    Shake it! detecting flaky tests caused by concurrency with shaker,

    D. Silva, L. Teixeira, and M. d’Amorim, “Shake it! detecting flaky tests caused by concurrency with shaker,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 301–311, IEEE, 2020

    2020

  8. [16]

    iDFlakies: A framework for detecting and partially classifying flaky tests,

    W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “iDFlakies: A framework for detecting and partially classifying flaky tests,” in ICST 2019: 12th IEEE International Conference on Software Testing, Verification and Validation, (Xi’an, China), pp. 312–322, April 2019

    2019

  9. [17]

    Empirically revisiting the test independence assumption,

    S. Zhang, D. Jalali, J. Wuttke, K. Muslu, W. Lam, M. D. Ernst, and D. Notkin, “Empirically revisiting the test independence assumption,” in International Symposium on Software Testing and Analysis, 2014

    2014

  10. [18]

    What is the vocabulary of flaky tests?,

    G. Pinto, B. Miranda, S. Dissanayake, M. d’Amorim, C. Treude, and A. Bertolino, “What is the vocabulary of flaky tests?,” in Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, (New York, NY, USA), p. 492–502, Association for Computing Ma...

    2020

  11. [19]

    Towards a bayesian netwo rk model for predicting flaky automated tests,

    T. M. King, D. Santiago, J. Phillips, and P. J. Clarke, “Towards a bayesian netwo rk model for predicting flaky automated tests,” in 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 100–107, IEEE, 2018

    2018

  12. [20]

    An empirical analysis of flaky tests,

    Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp. 643– 653, 2014

    2014

  13. [21]

    An empirical study of c++ vulnerabilities in crowd-sourced code examples,

    M. Verdi, A. Sami, J. Akhondali, F. Khomh, G. Uddin, and A. K. Motlagh, “An empirical study of c++ vulnerabilities in crowd-sourced code examples,” IEEE Transactions on Software Engineering, vol. 48, pp. 1497–1514, 2019

    2019

  14. [22]

    Enhancing large language models for text-to-testcase generation,

    S. Alagarsamy, C. Tantithamthavorn, C. Arora, and A. Aleti, “Enhancing large language models for text-to-testcase generation,” arXiv preprint arXiv:2402.11910, 2024

    2024 arXiv

  15. [23]

    Summary of chatgpt-related research and perspective towards the future of large language models,

    Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, et al., “Summary of chatgpt-related research and perspective towards the future of large language models,” Meta - Radiology, p. 100017, 2023

    2023

  16. [24]

    Codet5: Identifier -aware unified pre -trained encoderdecoder models for code understanding and generation,

    Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier -aware unified pre -trained encoderdecoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859, 2021

    2021 arXiv

  17. [25]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Pen g, “No more manual tests? evaluating and improving chatgpt for unit test generation,” arXiv preprint arXiv:2305.04207, 2023

    2023 arXiv

  18. [26]

    Chatunitest: a chatgpt -based automated unit test generation tool,

    Z. Xie, Y. Chen, C. Zhi, S. Deng, and J. Yin, “Chatunitest: a chatgpt -based automated unit test generation tool,” arXiv preprint arXiv:2305.04764, 2023

    2023 arXiv

  19. [27]

    Chapter 7 - learning to weight similarity measures with siamese networks: a case study on optimum -path forest,

    G. H. de Rosa and J. P. Papa, “Chapter 7 - learning to weight similarity measures with siamese networks: a case study on optimum -path forest,” in Optimum -Path Forest (A. X. Falc˜ao and J. P. Papa, eds.), pp. 155–173, Academic Press, 2022

    2022

  20. [28]

    Black -box prediction of flaky test fix categories using language models,

    S. Fatima, H. Hemmati, and L. Briand, “Black -box prediction of flaky test fix categories using language models,” arXiv preprint arXiv:2307.00012, 2023

    2023 arXiv

  21. [29]

    Data augmentation using pre -trained transformer models,

    V. Kumar, A. Choudhary, and E. Cho, “Data augmentation using pre -trained transformer models,” in Proceedings of the 2nd Workshop on Life -long Learning for Spoken Language Systems (W. M. Campbell, A. Waibel, D. Hakkani-Tur, T. J. Hazen, K. Kilgour, E. Cho, V. Kumar, and H. Gl...

    2020

  22. [30]

    Learning data manipulation for augmentation and weighting,

    Z. Hu, B. Tan, R. R. Salakhutdinov, T. M. Mitchell, and E. P. Xing, “Learning data manipulation for augmentation and weighting,” in Adva nces in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e -Buc, E. Fox, and R. Garnett, eds.), v...

    2019

  23. [31]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  24. [32]

    Smote: synthetic minority oversampling technique,

    N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority oversampling technique,” Journal of artificial intelligence research, vol. 16, pp. 321–357, 2002

    2002

  25. [33]

    Test flakiness across programming languages,

    K. Barbosa, R. Ferreira, G. Pinto, M. d’Amorim, and B. Miranda, “Test flakiness across programming languages,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2039 – 2052, 2023

    2023

  26. [34]

    What made this test flake? pinpointing classes responsible for test flakiness,

    S. Habchi, G. Haben, J. Sohn, A. Franci, M. Papadakis, M. Cordy, and Y. Traon, “What made this test flake? pinpointing classes responsible for test flakiness,” in 2022 IEEE International Conference Computer Science & Information Technology (CS & IT) 219 on Software Maintenance...

    2022

  27. [35]

    ifixflakies: a framework for automatically fixing order-dependent flaky tests,

    A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov, “ifixflakies: a framework for automatically fixing order-dependent flaky tests,” in Proceedings of the 2019 27th ACM Joint Meeting on Eu ropean Software Engineering Conference and Symposium on the Foundations of Software Engineer...

    2019

  28. [36]

    Rag vs fine -tuning: Pipelines, tradeoffs, and a case study on agriculture,

    A. Gupta, A. Shirgaonkar, A. d. L. Balaguer, B. Silva, D. Holstein, D. Li, J. Marsman, L. O. Nunes, M. Rouzbahman, M. Sharp, et al., “Rag vs fine -tuning: Pipelines, tradeoffs, and a case study on agriculture,” arXiv preprint arXiv:2401.08406, 2024

    2024 arXiv

  29. [37]

    Llama 2: Open foundation and fine -tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Ba baei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: Open foundation and fine -tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  30. [38]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressa nd, G. Lengyel, G. Lample, L. Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  31. [39]

    Lora: Low - rank adaptation of large language models,

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen -Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low - rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  32. [40]

    Label supervised llama finetuning,

    Z. Li, X. Li, Y. Liu, H. Xie, J. Li, F. -l. Wang, Q. Li, and X. Zhong, “Label supervised llama finetuning,” arXiv preprint arXiv:2310.01208, 2023

    2023 arXiv

  33. [41]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grise l, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learn...

    2011

  34. [42]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. and others, “Evaluating large language models trained on code,” 2021

    2021

  35. [43]

    Fine tuning vs. retrieval augmented generation for less popular knowledge,

    H. Soudani, E. Kanoulas, and F. Hasibi, “Fine tuning vs. retrieval augmented generation for less popular knowledge,” in Proceedings of the 202 4 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 12–22...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.