Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned GPT-4o sorts GitHub issues into bug/feature/question with 85.7% F1, and a tenfold increase in training data did not improve the score.

desk verdict Plausible cost/performance numbers and an honest discussion, but the 'more data didn't help' claim is confounded by dataset differences and needs deconfounding before it can be taken as a size effect. read the letter →

arxiv 2506.00128 v1 pith:N5NOV6ME submitted 2025-05-30 cs.SE cs.LG

Gabriel Aracena , Kyle Luster , Fabio Santos , Igor Steinmacher , Marco A. Gerosa This is my paper
classification cs.SEcs.LG
keywords issuereportclassificationlargelanguagemodelsfine-tuningGPT-4oGitHubissuesbugfeaturequestionNLBSEcost-performance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a fine-tuned GPT-4o model classifies GitHub issue reports into bugs, features, and questions with an average F1 score of 85.66% on the NLBSE 2024 competition dataset, beating both a fine-tuned GPT-3.5 model and the competition baseline. On a ten-times larger dataset from the NLBSE 2023 competition, the same fine-tuning recipe did not improve results: the F1 score fell to 80.7%, which the authors read as evidence that dataset quality matters more than dataset size. A smaller and cheaper model, GPT-4o-mini, matched GPT-4o's extended-dataset performance (80.37% F1) at roughly a tenth of the training cost, while a fine-tuned 8-billion-parameter DeepSeek-R1 variant lagged far behind (59.33% F1). The practical aim is to show that maintainers can get reliable automatic issue triage without assembling massive curated training sets. The paper's finding would matter because it shifts the bottleneck for automated issue labeling from data volume to data curation and model choice.

What carries the argument

The load-bearing mechanism is API fine-tuning of a pre-trained LLM on issue title and body text, with the label as the expected response, using a one-token constrained output and temperature zero. The central object is the fine-tuned GPT-4o model, evaluated by macro-averaged precision, recall, and F1 over three labels (bug, feature, question); the comparison across datasets isolates the effect of scaling training data. The extended dataset construction, taking the first 10,000 issues per label from a 1.4-million-issue pool and splitting them evenly into training and testing, is what carries the claim that tenfold data did not help. The GPT prompt format and the chain-of-thought, LoRA-based fine-tuning of DeepSeek are auxiliary machinery that explains why the GPT models and the DeepSeek variant behave differently.

What would settle it

Fine-tune GPT-4o on a 30,000-issue sample drawn from the same five repositories and the same curation pipeline as the NLBSE 2024 dataset. If the average F1 rises above 85.66%, the size effect is real and the paper's quality-over-quantity reading is wrong; if it stays at or below 80.7%, the conclusion is supported.

Watch

Extended reading notes

Core claim

The authors establish that, for the three-way classification of issue reports, fine-tuning a frontier general-purpose language model on a small curated corpus outperforms both traditional fine-tuned baselines and an open-weights reasoning model trained on ten times more data. Their central result is the average F1 of 85.66% achieved by fine-tuned GPT-4o on the NLBSE 2024 dataset, with per-repository precision reaching 98.82% for 'feature' labels in one project. They then show that moving to a 30,000-sample dataset from the NLBSE 2023 competition lowers the average F1 to 80.7% for GPT-4o, that GPT-4o-mini scores 80.37% at a tenth of the cost, and that DeepSeek-R1-Distill-Llama-8B scores only 59.33%. The conclusion they draw is that data quality, especially curation and label consistency, dominates dataset size when fine-tuning LLMs for this task.

Load-bearing premise

The 3,000-issue NLBSE 2024 dataset and the 30,000-issue NLBSE 2023 dataset are treated as comparable except for size, so the conclusion that more data does not help depends on ignoring differences in repositories, curation effort, label distributions, and issue ordering.

Editorial extensions

If this is right

  • Fine-tuned GPT-4o outperforms both the NLBSE 2024 baseline and fine-tuned GPT-3.5 on every label and metric, so upgrading the base model is a reliable way to improve automated triage.
  • A tenfold increase in fine-tuning data, from 3,000 to 30,000 issues, did not raise F1; therefore practitioners should prioritize curated, label-consistent datasets over raw volume.
  • GPT-4o-mini's near-identical extended-dataset results at roughly one-tenth the cost make it the recommended model for cost-sensitive issue-classification pipelines.
  • DeepSeek-R1-Distill-Llama-8B, as configured here, is not competitive for this three-way classification, so its cost advantage does not translate into useful accuracy.
  • The 'question' label is the hardest class across repositories, so improvements in detecting implicit questions would yield the largest gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'dataset size does not help' conclusion is confounded: the 3,000-sample and 30,000-sample sets differ in repositories, curation effort, label balance, and issue ordering, so a direct causal reading requires holding those factors fixed; a cleaner test would fine-tune on a 30,000-sample set drawn from the same five repositories used in NLBSE 2024.
  • If data quality is the dominant factor, then a small, actively curated set of issues from each repository should outperform a generic large pool; this is testable per repository and would give maintainers a cheap procurement rule.
  • The cost result suggests a practical deployment recipe: use a small curated sample and a mid-size model, and only scale data after checking label consistency, which reverses the usual 'more data wins' assumption in supervised learning.
  • The one-token constrained output and temperature zero make the evaluation deterministic; a natural extension is to add a confidence threshold so ambiguous 'question' cases are routed to human triage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates fine-tuned large language models for GitHub issue classification into bug, feature, and question categories. Using the NLBSE 2024 competition dataset (3,000 curated, balanced issues from five repositories), the authors fine-tune GPT-3.5 and GPT-4o, reporting average F1 scores of 82.8% and 85.66%, respectively, with the GPT-4o result exceeding the competition baseline. Using a 30,000-issue subset of the NLBSE 2023 dataset, they fine-tune GPT-4o, GPT-4o-mini, and DeepSeek-R1-Distill-Llama-8B, reporting F1 scores of 80.7%, 80.37%, and 59.33%. The authors interpret the 30k results as evidence that increasing dataset size does not improve F1 and that data quality matters more than quantity, and they conclude that GPT-4o-mini offers the best cost-performance trade-off. The paper also compares vanilla and fine-tuned GPT-4o, finding fine-tuning provides substantial gains.

Significance. If taken as a benchmark across fixed public datasets, the study provides useful, reproducible evidence: fine-tuned GPT-4o outperforms a strong sentence-transformer baseline on the small curated NLBSE 2024 benchmark, and GPT-4o-mini achieves nearly the same F1 at roughly one-tenth the fine-tuning cost. The paper also ships an open repository link and uses standard scikit-learn metrics, which supports external verification. The broad conclusion about dataset size, however, is not supported by the experimental design because the 3k and 30k cells differ in curation, repository composition, label set, and test set. The overall significance is therefore moderate: the paper is a solid benchmark contribution for RQ1/RQ2 but its headline RQ3 claim about scale needs substantial reframing or additional controlled experiments.

major comments (4)
  1. [Sections 3.1.1, 3.1.4, and 5] The central RQ3 claim that a 10x larger dataset does not improve F1 is confounded. The NLBSE 2024 cell uses a carefully curated, balanced 3k sample from five repositories with a 1.5k test set, while the NLBSE 2023 cell uses the first 10,000 records per label from a 1.4M-issue pool, different repositories, and drops the documentation label. Section 5 explicitly concedes that NLBSE 2024 was carefully curated and NLBSE 2023 was not subject to the same rigorous selection. The observed drop from 85.66% to 80.70% can therefore be attributed to curation, repository mix, label noise, or test-set difficulty, not to dataset size. The abstract's statement that 'increasing dataset size did not improve the F1 score' is a causal reading the design cannot support and should be removed or reframed as a comparison of two datasets that differ in multiple ways.
  2. [Section 3.1.3 and Section 4 (RQ1)] The per-repository cleaning method was selected after inspecting test-set performance: Method 2 was retained where it improved results and Method 1 was kept for two repositories. This post hoc selection makes the reported F1 values optimistically biased estimates of the proposed cleaning pipelines. The paper should describe this as an exploratory selection, report both methods for all five repositories, or validate the choice on a held-out split; otherwise the headline 82.8% and 85.66% figures are not unbiased estimates of a predefined method.
  3. [Section 4, Tables 1-6] No confidence intervals, error bars, or significance tests are reported. The text repeatedly calls differences 'significant' (for example, the GPT-3.5 to GPT-4o improvement in Section 4), and the RQ3 discussion relies on small aggregate gaps (80.7% vs. 80.37%) without uncertainty quantification. Since each reported number comes from a single API fine-tuning run, the authors should provide at least bootstrap confidence intervals or repeated runs for the headline comparisons before making strong claims about improvement or lack thereof.
  4. [Sections 3.2.1, 3.2.3, and 5] The GPT-4o versus DeepSeek-R1 comparison is not controlled: GPT models are fine-tuned through OpenAI's API with the exact conversation prompt, while DeepSeek uses an Unsloth/LoRA setup, a different prompt style with explicit chain-of-thought, quantization, and different hyperparameters. The conclusion that 'DeepSeek has not shown to be a viable alternative' (Section 8) is only valid for this particular implementation, not for 'both models trained on the same dataset' as claimed. The paper should narrow the claim to the specific fine-tuning configurations used.
minor comments (5)
  1. [Table 2] Several NLBSE 2024 baseline F1 values are inconsistent with the reported precision and recall values: for feature, precision 0.8448 and recall 0.8700 imply F1 ≈ 0.857, not 0.8426; for question, 0.8001 and 0.7700 imply F1 ≈ 0.785, not 0.7827. Please verify the baseline metrics and correct the table.
  2. [Section 5, 'What is the impact of the extended dataset?'] The sentence claiming a 'magnificent 94.8%' question precision from the extended dataset and attributing it to Table 5 is incorrect: Table 5 reports question precision of 76.52% for GPT-4o and 77.13% for GPT-4o-mini. The 94.8% value appears to be the facebook/react question precision in Table 4, which is from the NLBSE 2024 dataset, not the extended dataset.
  3. [Table 3] The GPT-4o vanilla overall recall is printed as '06860'; this should be '0.6860', and the formatting should match the other numeric entries.
  4. [Throughout] The spelling of the fine-tuning library is inconsistent: 'UnsLoTH' appears in Sections 3.2.1 and 3.2.3, while the standard spelling 'Unsloth' is used elsewhere. Please standardize the spelling and the acronym usage (for example, 'LoRA' rather than 'Low-Rank Adaptation (LoRA) adaptation').
  5. [Table 9] The FastText and RoBERTa average rows list identical values for precision, recall, and F1 (0.8510 and 0.8906). These values fail the usual F1 relation with precision and recall and should be recomputed from the per-label confusion matrices or replaced with actual averages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a self-contained empirical benchmark whose results are judged against external NLBSE baselines; the only self-citation is a non-load-bearing starting point.

full rationale

The paper's derivation chain is empirical rather than formal: it fine-tunes models on public NLBSE datasets, evaluates on held-out test splits, and computes precision, recall, and F1 from model predictions against ground-truth labels using sklearn. No equation reduces a predicted quantity to a fitted parameter, and no target result is embedded in the definition of an input. The strongest candidate for concern is RQ3, where the 3k NLBSE 2024 cell and the 30k NLBSE 2023 cell differ in curation, repositories, label handling, and test set; Section 5 explicitly concedes that the NLBSE 2024 dataset was 'carefully curated' while the NLBSE 2023 dataset 'was not subject to the same rigorous selection process.' This is a validity confound that weakens the causal reading that 'dataset size did not improve F1', but it is not circularity: the compared runs are independent experimental cells, not outputs defined by their inputs. The paper's self-citation [6] is used only as prior work being extended, not as evidence for the current results, and the central comparisons are anchored to external competition baselines ([19] and NLBSE 2023 baselines). No self-definitional step, fitted-input-called-prediction step, or self-citation chain carries the conclusions. Accordingly, the appropriate finding is no significant circularity, with the RQ3 interpretation flagged as an internal-validity limitation rather than a circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The results rest on benchmark datasets and API pipelines; the main free choices are the post hoc cleaning-method selection, epoch grid search, and the ordered subset from NLBSE 2023. No new theoretical entities are introduced.

free parameters (4)
  • Cleaning method assignment per repository = facebook: Method 1, tensorflow: Method 2, microsoft: Method 1, bitcoin: Method 1, opencv: Method 2 (Table 1)
    Chosen post hoc based on which method yielded better F1 on the test set, so reported performance is selected rather than predicted.
  • Fine-tuning epochs per repository (GPT-3.5) = facebook 3, tensorflow 10, microsoft 6, bitcoin 3, opencv 6
    Grid-searched to optimize training metrics; not fixed a priori. GPT-4o epoch counts are not reported.
  • NLBSE 2023 subset order and size = 30,000 (first 10,000 per label, in file order)
    The first issues per label are used, not a random sample, which may bias the dataset and the resulting performance comparison.
  • LoRA rank and alpha for DeepSeek = rank 16, alpha 16
    Hyperparameters chosen for the DeepSeek pipeline without a reported search; specific to that model's LoRA configuration.
assumptions (4)
  • domain assumption The ground-truth labels in the NLBSE datasets are correct enough to serve as evaluation targets.
    The paper itself gives examples of mislabeled 'question' issues in Section 5, so label noise is present and may affect metrics.
  • ad hoc to paper Fine-tuning via OpenAI's API and via Unsloth/LoRA are comparable instantiations of 'fine-tuning an LLM'.
    The two pipelines differ in prompt format, optimization strategy, and base model access, so differences in F1 cannot be cleanly attributed to the model.
  • domain assumption The NLBSE 2024 and NLBSE 2023 datasets are comparable enough to infer the effect of dataset size.
    They differ in repositories, label sets, curation, and collection period; the paper interprets the F1 drop as a dataset-quality effect rather than evidence about size alone.
  • domain assumption API temperature 0.0 and max_tokens=1 produce deterministic and parseable outputs.
    Used to justify single-run evaluations; not independently verified for all models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models." pith.science (2026). https://pith.science/paper/N5NOV6ME

@misc{pith2026250600128,
  author       = {Pith},
  title        = {Pith review of: Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N5NOV6ME}},
  note         = {Machine review of arXiv:2506.00128}
}
read the original abstract

Effective prioritization of issue reports in software engineering helps to optimize resource allocation and information recovery. However, manual issue classification is laborious and lacks scalability. As an alternative, many open source software (OSS) projects employ automated processes for this task, yet this method often relies on large datasets for adequate training. Traditionally, machine learning techniques have been used for issue classification. More recently, large language models (LLMs) have emerged as powerful tools for addressing a range of software engineering challenges, including code and test generation, mapping new requirements to legacy software endpoints, and conducting code reviews. The following research investigates an automated approach to issue classification based on LLMs. By leveraging the capabilities of such models, we aim to develop a robust system for prioritizing issue reports, mitigating the necessity for extensive training data while also maintaining reliability in classification. In our research, we developed an LLM-based approach for accurately labeling issues by selecting two of the most prominent large language models. We then compared their performance across multiple datasets. Our findings show that GPT-4o achieved the best results in classifying issues from the NLBSE 2024 competition. Moreover, GPT-4o outperformed DeepSeek R1, achieving an F1 score 20% higher when both models were trained on the same dataset from the NLBSE 2023 competition, which was ten times larger than the NLBSE 2024 dataset. The fine-tuned GPT-4o model attained an average F1 score of 80.7%, while the fine-tuned DeepSeek R1 model achieved 59.33%. Increasing the dataset size did not improve the F1 score, reducing the dependence on massive datasets for building an efficient solution to issue classification.

Figures

Figures reproduced from arXiv: 2506.00128 by the authors.

Figure 1
Figure 1. Method. Fine-tuned (FT) models and Zero-shot (ZS) Models. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 49 canonical work pages

  1. [1]

    Ahmed Adnan, Antu Saha, and Oscar Chaparro. 2025. SPRINT: An As- sistant for Issue Report Management. arXiv preprint arXiv:2502.04147 (2025)

    work page Pith review arXiv 2025

  2. [2]

    Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards human-bot collaborative software architecting with chatgpt. In Proceedings of the 29 27th International Conference on Evaluation and Assessment in Soft- ware Engineering. 279–285

    work page 2023

  3. [3]

    Giuliano Antoniol, Kamel Ayari, Massimiliano Di Penta, Foutse Khomh, and Yann-Ga¨ el Gu´ eh´ eneuc. 2008. Is it a bug or an enhancement? A text- based approach to classify change requests. In Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds . 304–318

    work page 2008

  4. [4]

    John Anvik, Lyndon Hiew, and Gail C Murphy. 2006. Who should fix this bug?. In Proceedings of the 28th international conference on Software engineering. 361–370

    2006

  5. [5]

    John Anvik and Gail C Murphy. 2011. Reducing the effort of bug re- port triage: Recommenders for development-oriented decisions. ACM Transactions on Software Engineering and Methodology (TOSEM) 20, 3 (2011), 10

    work page 2011

  6. [6]

    Gabriel Aracena, Kyle Luster, Fabio Santos, Igor Steinmacher, and Marco Aurelio Gerosa. 2024. Applying large language models to is- sue classification. In Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering . 57–60

    work page 2024

  7. [7]

    Barcomb, K

    A. Barcomb, K. Stol, B. Fitzgerald, and D. Riehle. 2020. Managing Episodic Volunteers in Free/Libre/Open Source Software Communities. IEEE Transactions on Software Engineering (2020), 1–1

    work page 2020

  8. [8]

    Benjamin C Carter, Jonathan Rivas Contreras, Carlos A Llanes Villegas, Pawan Acharya, Jack Utzerath, Adonijah O Farner, Hunter Jenkins, Dylan Johnson, Jacob Penney, Igor Steinmacher, et al. 2025. SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub. In Proceedings of the Fourth ACM/IEEE International Workshop on NL-based Soft...

    work page 2025

Show all 56 references
  1. [9]

    Giuseppe Colavito, Filippo Lanubile, and Nicole Novielli. 2023. Few- Shot Learning for Issue Report Classification. In 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineer- ing (NLBSE) . 16–19. https://doi.org/10.1109/NLBSE59153.2023. 00011

    2023

  2. [10]

    Giuseppe Colavito, Filippo Lanubile, Nicole Novielli, and Luigi Quar- anta. 2024. Impact of data quality for automatic issue classification using pre-trained language models. Journal of Systems and Software 210 (2024), 111838. 30

    2024

  3. [11]

    Giuseppe Colavito, Filippo Lanubile, Nicole Novielli, and Luigi Quar- anta. 2024. Leveraging gpt-like llms to automate issue labeling. In Pro- ceedings of the 21st International Conference on Mining Software Repos- itories. 469–480

    2024

  4. [12]

    Eleni Constantinou and Tom Mens. 2017. An empirical comparison of developer retention in the RubyGems and npm software ecosystems. Innovations in Systems and Software Engineering 13 (2017), 101–115

    2017

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  6. [14]

    Farida El Zanaty, Christophe Rezk, Sander Lijbrink, Willem van Bergen, Mark Cˆ ot´ e, and Shane McIntosh. 2020. Automatic Recovery of Missing Issue Type Labels. IEEE Software (2020)

    2020

  7. [15]

    Yuekai Huang, Junjie Wang, Song Wang, Zhe Liu, Dandan Wang, and Qing Wang. 2021. Characterizing and Predicting Good First Issues. In Proceedings of the 15th ACM/IEEE International Symposium on Em- pirical Software Engineering and Measurement (ESEM) . 1–12

    2021

  8. [16]

    Maliheh Izadi, Kiana Akbari, and Abbas Heydarnoori. 2022. Predict- ing the objective and priority of issue reports in software repositories. Empirical Software Engineering 27, 2 (2022), 1–37

    2022

  9. [17]

    Maliheh Izadi, Siavash Ganji, and Abbas Heydarnoori. 2021. Topic Rec- ommendation for Software Repositories using Multi-label Classification Algorithms. Empir. Softw. Eng. 26 (2021), 93

    2021

  10. [18]

    Maliheh Izadi, Mahtab Nejati, and Abbas Heydarnoori. 2023. Semantically-enhanced topic recommendation systems for software projects. Empirical Software Engineering 28, 2 (2023), 50

    2023

  11. [19]

    Rafael Kallis, Giuseppe Colavito, Ali Al-Kaswan, Luca Pascarella, Oscar Chaparro, and Pooja Rani. 2024. The NLBSE’24 Tool Competition. In Proceedings of The 3rd International Workshop on Natural Language- based Software Engineering (NLBSE’24)

    2024

  12. [20]

    Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2019. Ticket Tagger: Machine Learning Driven Issue Clas- sification. In 2019 IEEE International Conference on Software Mainte- nance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 31 - Oct...

    2019

  13. [21]

    Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2019. Ticket tagger: Machine learning driven issue classifica- tion. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 406–409

    2019

  14. [22]

    Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2021. Predicting issue types on GitHub. Science of Com- puter Programming 205 (2021), 102598. https://doi.org/10.1016/ j.scico.2020.102598

    2021

  15. [23]

    Rafael Kallis, Maliheh Izadi, Luca Pascarella, Oscar Chaparro, and Pooja Rani. 2023. The NLBSE’23 Tool Competition. In Proceedings of The 2nd International Workshop on Natural Language-based Software Engineering (NLBSE’23)

    2023

  16. [24]

    Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, and Michele Lanza. 2019. Pattern-based mining of opinions in Q&A websites. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 548–559

    2019

  17. [25]

    Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland- Huang. 2021. Traceability transformed: Generating more accurate links with pre-trained bert models. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 324–335

    2021

  18. [26]

    Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020. Multi-task learning based pre-trained language model for code completion. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 473–485

    2020

  19. [27]

    Courtney Miller, David Gray Widder, Christian K¨ astner, and Bogdan Vasilescu. 2019. Why Do People Give Up FLOSSing? A Study of Con- tributor Disengagement in Open Source. In Open Source Systems, Fran- cis Bordeleau, Alberto Sillitti, Paulo Meirelles, and Valentina Lenarduzzi ...

    2019

  20. [28]

    Georges A Miller, Edwin Broomell Newman, and Elizabeth A Friedman

  21. [29]

    Lidia PG Nascimento, Alcemir Santos, Ivan Machado, et al. 2024. Issue Labeling Dynamics in Open-Source Projects: A Comprehensive Analy- sis. In Simp´ osio Brasileiro de Componentes, Arquiteturas e Reutiliza¸ c˜ ao de Software (SBCARS) . SBC, 51–60

    2024

  22. [30]

    Nascimento Nathalia, Alencar Paulo, and Cowan Donald. 2023. Artifi- cial Intelligence vs. Software Engineers: An Empirical Study on Perfor- mance and Efficiency using ChatGPT. InProceedings of the 33rd Annual International Conference on Computer Science and Software Engineer- ...

    2023

  23. [31]

    Ipek Ozkaya. 2023. Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications. IEEE Soft- ware 40, 3 (2023), 4–8

    2023

  24. [32]

    Quentin Perez, Pierre-Antoine Jean, Christelle Urtado, and Sylvain Vauttier. 2021. Bug or not bug? That is the question. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 47–58

    2021

  25. [33]

    Natthakul Pingclasai, Hideaki Hata, and Ken-ichi Matsumoto. 2013. Classifying bug reports to bugs and other requests using topic modeling. In 2013 20Th asia-pacific software engineering conference (APSEC) , Vol. 2. IEEE, 13–18

    2013

  26. [34]

    Fabio Santos, Jacob Penney, Jo˜ ao Felipe Pimentel, Igor Wiese, Bianca Trinkenreich, Igor Steinmacher, and Marco A Gerosa. 2023. Tell Me Who Are You Talking to and I Will Tell You What Issues Need Your Skills. In 2023 IEEE/ACM 20th International Conference on Mining Software R...

    2023

  27. [35]

    Fabio Santos, Bianca Trinkenreich, Jo˜ ao Felipe Nicolati Pimentel, Igor Wiese, Igor Steinmacher, Anita Sarma, and Marco Gerosa. 2022. How to choose a task? Mismatches in perspectives of newcomers and existing contributors. Empirical Software Engineering and Measurement (2022)

    2022

  28. [36]

    Fabio Santos, Joseph Vargovich, Bianca Trinkenreich, Italo Santos, Ja- cob Penney, Ricardo Britto, Jo˜ ao Felipe Pimentel, Igor Wiese, Igor Steinmacher, Anita Sarma, et al. 2023. Tag that issue: applying API- domain labels in issue tracking systems. Empirical Software Engineer...

    2023

  29. [37]

    Fabio Santos, Igor Wiese, Bianca Trinkenreich, Igor Steinmacher, Anita Sarma, and Marco Gerosa. 2021. Can I Solve It? Identifying APIs Required to Complete OSS Task. (2021)

    2021

  30. [38]

    Christoph Stanik, Lloyd Montgomery, Daniel Martens, Davide Fucci, and Walid Maalej. 2018. A Simple NLP-based Approach to Support Onboarding and Retention in Open Source Communities. In 2018 IEEE International Conference on Software Maintenance and Evolution (IC- SME). IEEE, 172–182

    2018

  31. [39]

    Igor Steinmacher, Tayana Conte, Marco Aur´ elio Gerosa, and David Red- miles. 2015. Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & So- cial...

    2015

  32. [40]

    Igor Steinmacher, Tayana Uchˆ oa Conte, and Marco Aur´ elio Gerosa

  33. [41]

    Igor Steinmacher, Gustavo Pinto, Igor Scaliante Wiese, and Marco A Gerosa. 2018. Almost there: A study on quasi-contributors in open source software projects. In Proceedings of the 40th international con- ference on software engineering . 256–266

    2018

  34. [42]

    Igor Steinmacher, Marco Aurelio Graciotto Silva, Marco Aurelio Gerosa, and David F Redmiles. 2015. A systematic literature review on the bar- riers faced by newcomers to open source software projects. Information and Software Technology 59 (2015), 67–85

    2015

  35. [43]

    Igor Steinmacher, Christoph Treude, and Marco Aurelio Gerosa. 2018. Let me in: Guidelines for the successful onboarding of newcomers to open source projects. IEEE Software 36, 4 (2018), 41–49

    2018

  36. [44]

    Igor Steinmacher, Igor Wiese, Ana Paula Chaves, and Marco Aur´ elio Gerosa. 2013. Why do newcomers abandon open source software projects?. In 2013 6th International Workshop on Cooperative and Hu- man Aspects of Software Engineering (CHASE) . IEEE, 25–32

    2013

  37. [45]

    Gias Uddin and Foutse Khomh. 2019. Automatic mining of opinions expressed about apis in stack overflow. IEEE Transactions on Software Engineering 47, 3 (2019), 522–559. 34

    2019

  38. [46]

    Santiago Vargas-Baldrich, Mario Linares-V´ asquez, and Denys Poshy- vanyk. 2015. Automated tagging of software projects using bytecode and dependencies. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 289–294

    2015

  39. [47]

    Joseph Vargovich, Fabio Santos, Jacob Penney, Marco Aur´ elio Gerosa, and Igor Steinmacher. 2023. GiveMeLabeledIssues: An Open Source Issue Recommendation System. In 20th IEEE/ACM International Con- ference on Mining Software Repositories, MSR 2023, Melbourne, Aus- tralia, May...

    2023

  40. [48]

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023. DecodingTrust: A Comprehensive Assessment of Trustwor- thiness in GPT Models.. In NeurIPS

    2023

  41. [49]

    Jianguo Wang and Anita Sarma. 2011. Which bug should I fix: help- ing new developers onboard a new project. In Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering. ACM, 76–79

    2011

  42. [50]

    Jun Wang, Xiaofang Zhang, and Lin Chen. 2021. How well do pre-trained contextual language representations recommend labels for GitHub issues? Knowledge-Based Systems 232 (2021), 107476. https: //doi.org/10.1016/j.knosys.2021.107476

    2021

  43. [51]

    Xin Xia, David Lo, Xinyu Wang, and Bo Zhou. 2013. Tag recommen- dation in software information sites. In 2013 10th Working Conference on Mining Software Repositories (MSR) . IEEE, 287–296

    2013

  44. [52]

    Yu Zhou, Yanxiang Tong, Ruihang Gu, and Harald Gall. 2016. Combin- ing text mining and data mining for bug report classification. Journal of Software: Evolution and Process 28, 3 (2016), 150–176

    2016

  45. [53]

    Yuxiang Zhu, Minxue Pan, Yu Pei, and Tian Zhang. 2019. A Bug or a Suggestion? An Automatic Way to Label Issues. arXiv preprint arXiv:1909.00934 (2019). 35

    2019 arXiv

  46. [1958]

    Information and control 1, 4 (1958), 370–389

    Length-frequency statistics for written English. Information and control 1, 4 (1958), 370–389. 32

    1958

  47. [2015]

    In 2015 48th Hawaii International Conference on System Sciences

    Understanding and supporting the choice of an appropriate task to start with in open source software communities. In 2015 48th Hawaii International Conference on System Sciences . IEEE, 5299–5308

    2015

  48. [2019]

    Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North Amer- ican chapter of the association for computational linguistics: human lan- guage technologies, volume 1 (long and short papers) . 4171–4186

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.