pith. sign in

arxiv: 2606.11476 · v1 · pith:CS2RBNHUnew · submitted 2026-06-09 · 💻 cs.SE

SentTrack: Sentiment-Driven Bottleneck Detection in GitHub Issue Repositories

Xinyu Hu , Ali Behbahani , Daniel Moon , Yaren Dogan , Nasir U. Eisty This is my paper

Pith reviewed 2026-06-27 12:02 UTC · model grok-4.3

classification 💻 cs.SE
keywords GitHub issuessentiment analysisbottleneck detectionissue trackingsocio-technical systemscollaborative developmentworkflow efficiency
0
0 comments X

The pith

SentTrack's weighted scoring engine uses conversation signals to prioritize high-friction GitHub issue threads before they stall.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SentTrack, a dual-lens framework that processes GitHub issue threads to detect socio-technical bottlenecks. A horizontal pipeline turns raw reports into LLM summaries, extracts phrases, and clusters them with UMAP and HDBSCAN, while a vertical pipeline classifies comments via the ABCDE framework to determine thread outcomes. Applied to about 9,000 threads in one repository, the analysis finds 49% end in stagnation and only 13% reach resolution, with resolution gap as the strongest signal. The resulting weighted scoring engine combines negativity, stagnation, resolution gap, and thread length into an interpretable prioritization tool for maintainers.

Core claim

SentTrack demonstrates that automated analysis of conversational dynamics in GitHub issues can surface workflow inefficiencies, with the resolution gap identified as the dominant bottleneck and a composite score providing a practical way to rank threads by risk of stalling development.

What carries the argument

The weighted scoring engine that combines negativity, stagnation, resolution gap, and thread length, derived from LLM summaries, semantic clusters, and ABCDE interaction classifications.

If this is right

  • Maintainers gain an interpretable prioritization tool for high-friction discussions before they stall development.
  • Resolution gap is identified as the dominant bottleneck signal across the analyzed threads.
  • 49% of threads ended in stagnation while only 13% reached resolution.
  • The framework automates workflow-inefficiency detection from real-time conversational data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same dual-pipeline method could be applied to issue data from other platforms to check for comparable stagnation rates.
  • Embedding the scoring engine into repository dashboards might allow proactive alerts for at-risk discussions.
  • The separation of human narrative from machine-generated noise could extend to other mixed-content discussion systems.

Load-bearing premise

The assumption that LLM summaries, clusters, and ABCDE classifications accurately reflect real socio-technical bottlenecks rather than artifacts of the processing methods.

What would settle it

A controlled comparison that measures whether SentTrack scores predict actual stalled or high-friction threads more accurately than existing GitHub labels on the same corpus.

Figures

Figures reproduced from arXiv: 2606.11476 by Ali Behbahani, Daniel Moon, Nasir U. Eisty, Xinyu Hu, Yaren Dogan.

Figure 1
Figure 1. Figure 1: Discovery rates of unique LLM topics and HDBSCAN clusters across [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Discovery rates of unique LLM topics and HDBSCAN clusters across [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative horizontal pipeline execution showing: (a) LLM noise [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Interaction transition heatmap for closed issue threads. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Interaction transition heatmap for open issue threads. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pipeline diagnostic metrics over the combined open and closed corpus. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Top-ranked issues by bottleneck score across open and closed threads. [PITH_FULL_IMAGE:figures/full_fig_p007_11.png] view at source ↗
Figure 9
Figure 9. Figure 9: Issue negativity score versus resolution status, colored by bottleneck [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
read the original abstract

Software engineering teams increasingly depend on GitHub issue threads to coordinate work, report bugs, and negotiate technical decisions, yet most repository health tools focus on code metrics and ignore the conversational dynamics that drive or stall development. This paper presents SentTrack, a dual-lens framework for detecting socio-technical bottlenecks from GitHub issue discussions. Applied to the AvaloniaUI open-source repository across approximately 9,000 issue threads, the framework addresses three questions: how to automate workflow-inefficiency detection from real-time conversational data, whether sentiment signals can surface risk earlier than traditional label-based methods, and how to isolate human narrative from machine-generated noise in mixed-media issue text. SentTrack combines two complementary pipelines. A horizontal pipeline translates raw issue reports into clean summaries using a large language model, extracts mid-level concern phrases, and clusters them through UMAP and HDBSCAN, producing 613 semantic clusters from the first 3,608 issues processed. A vertical pipeline applies the ABCDE collaborative interaction framework to classify each comment and infer thread-level outcomes. Across the full corpus, 49\% of threads ended in stagnation and only 13\% reached resolution, with the resolution gap identified as the dominant bottleneck signal. A weighted scoring engine that combines negativity, stagnation, resolution gap, and thread length gives maintainers an interpretable prioritization tool for high-friction discussions before they stall development.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper presents SentTrack, a dual-lens framework that uses LLM summarization of GitHub issue threads, UMAP+HDBSCAN clustering to produce 613 semantic clusters from 3608 issues, and ABCDE classification to detect socio-technical bottlenecks. Applied to the AvaloniaUI repository, it reports that 49% of threads stagnate while only 13% reach resolution, identifies the resolution gap as the dominant bottleneck, and introduces a weighted scoring engine combining negativity, stagnation, resolution gap, and thread length to prioritize high-friction discussions.

Significance. If the pipeline were shown to accurately map to real bottlenecks and outperform label-based methods, the approach could provide maintainers with an interpretable, conversation-aware prioritization tool that addresses a gap in repository health analysis beyond code metrics.

major comments (3)
  1. [Abstract] Abstract: the 49% stagnation and 13% resolution statistics are presented without any validation metrics, baselines, error rates, or description of how the figures were computed from the corpus or clusters, leaving the central claim that the framework detects bottlenecks unsupported.
  2. [Abstract] Abstract: the weighted scoring engine is claimed to yield an interpretable prioritization tool, yet the weights are free parameters with no account of their selection, tuning process, or whether they were derived from the same data used to report the stagnation statistics, creating a circularity risk.
  3. [Abstract] Abstract: the question of whether sentiment signals surface risk earlier than traditional label-based methods is explicitly left open, but the manuscript supplies no held-out evaluation, inter-rater agreement scores, or outcome-prediction comparison (e.g., ranking resolved vs. stagnant threads) to test this premise.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We agree that the abstract requires additional methodological detail and validation to support the reported statistics and claims. We will revise the manuscript accordingly by expanding the abstract, adding validation subsections, and including sensitivity analyses.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the 49% stagnation and 13% resolution statistics are presented without any validation metrics, baselines, error rates, or description of how the figures were computed from the corpus or clusters, leaving the central claim that the framework detects bottlenecks unsupported.

    Authors: We agree that the abstract omits these details. The full manuscript describes the vertical ABCDE pipeline for classifying thread outcomes, but does not report validation. We will revise by adding a methods subsection on classification definitions, computation from the 3608-issue sample, and inter-rater agreement plus error rates from a manually annotated subset. This will directly support the bottleneck claim. revision: yes

  2. Referee: [Abstract] Abstract: the weighted scoring engine is claimed to yield an interpretable prioritization tool, yet the weights are free parameters with no account of their selection, tuning process, or whether they were derived from the same data used to report the stagnation statistics, creating a circularity risk.

    Authors: The referee is correct that the current text provides no account of weight selection or tuning. The stagnation statistics derive solely from ABCDE classification and are independent of the scoring engine. In revision we will document the heuristic rationale drawn from prior literature, add a sensitivity analysis across weight ranges, and explicitly state the separation of data sources to remove any circularity. revision: yes

  3. Referee: [Abstract] Abstract: the question of whether sentiment signals surface risk earlier than traditional label-based methods is explicitly left open, but the manuscript supplies no held-out evaluation, inter-rater agreement scores, or outcome-prediction comparison (e.g., ranking resolved vs. stagnant threads) to test this premise.

    Authors: The manuscript intentionally leaves the comparative question open to focus on framework introduction. However, we accept that this leaves the premise untested. We will add a new evaluation section reporting inter-rater agreement for ABCDE labels and a held-out comparison of the scoring engine versus label-based baselines on resolution prediction, using ranking metrics such as AUC. revision: yes

Circularity Check

0 steps flagged

No circularity: descriptive pipeline with direct corpus counts

full rationale

The paper describes an application of LLM summarization, UMAP+HDBSCAN clustering (producing 613 clusters from 3608 issues), and ABCDE classification to extract features like stagnation (49%) and resolution (13%) directly from the AvaloniaUI corpus. The weighted scoring engine is presented as a linear combination of those extracted signals (negativity, stagnation, resolution gap, thread length) without any equations, fitting procedure, or self-citation that would make the scores or 'predictions' equivalent to the inputs by construction. No load-bearing uniqueness theorem, ansatz smuggling, or renaming of known results appears. The derivation chain consists of standard external techniques applied to issue text and remains self-contained.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 1 invented entities

Review performed on abstract only; full methods, validation, and parameter choices are unavailable, so the ledger is necessarily incomplete and conservative.

free parameters (1)
  • weights in scoring engine
    The engine combines negativity, stagnation, resolution gap, and thread length; specific weights are not stated and may have been set by hand or fit to the data.
axioms (2)
  • domain assumption UMAP and HDBSCAN produce semantically meaningful clusters from LLM-generated issue summaries
    Invoked to produce 613 clusters from the first 3,608 issues
  • domain assumption The ABCDE collaborative interaction framework can be applied to classify individual comments and infer thread-level outcomes
    Used in the vertical pipeline to classify each comment
invented entities (1)
  • SentTrack dual-lens framework no independent evidence
    purpose: Detect socio-technical bottlenecks from GitHub issue discussions
    New named system introduced by the paper

pith-pipeline@v0.9.1-grok · 5789 in / 1552 out tokens · 32371 ms · 2026-06-27T12:02:49.544455+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    GitHub,About Issues, GitHub Docs, n.d., accessed: Mar. 6,

  2. [2]

    Available: https://docs.github.com/en/issues/tracking- your-work-with-issues/learning-about-issues/about-issues

    [Online]. Available: https://docs.github.com/en/issues/tracking- your-work-with-issues/learning-about-issues/about-issues

  3. [3]

    ——,About Discussions, GitHub Docs, n.d., accessed: Mar. 6, 2026. [Online]. Avail- able: https://docs.github.com/en/discussions/collaborating-with-your- community-using-discussions/about-discussions

    2026

  4. [4]

    Opinion mining and sentiment analysis,

    B. Pang and L. Lee, “Opinion mining and sentiment analysis,”Founda- tions and Trends in Information Retrieval, vol. 2, no. 1–2, pp. 1–135, 2008

    2008

  5. [5]

    Liu,Sentiment Analysis and Opinion Mining, ser

    B. Liu,Sentiment Analysis and Opinion Mining, ser. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, 2012

    2012

  6. [6]

    Sentiment analysis in monitoring software development processes: An exploratory case study on GitHub’s project issues,

    F. Jurado and P. Rodriguez, “Sentiment analysis in monitoring software development processes: An exploratory case study on GitHub’s project issues,”Journal of Systems and Software, vol. 104, pp. 82–89, 2015

    2015

  7. [7]

    Sentiment analysis of commit comments in GitHub: An empirical study,

    E. Guzman, D. Az ´ocar, and Y . Li, “Sentiment analysis of commit comments in GitHub: An empirical study,” inProceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014, pp. 352–355

    2014

  8. [8]

    Security and emotion: Sentiment analysis of security discussions on GitHub,

    D. Pletea, B. Vasilescu, and A. Serebrenik, “Security and emotion: Sentiment analysis of security discussions on GitHub,” inProceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014, pp. 348–351

    2014

  9. [9]

    Are bullies more productive? empirical study of affective- ness vs. issue fixing time,

    M. Ortu, B. Adams, G. Destefanis, P. Tourani, M. Marchesi, and R. Tonelli, “Are bullies more productive? empirical study of affective- ness vs. issue fixing time,” in2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 303–313

    2015

  10. [10]

    The emotional side of software developers in JIRA,

    M. Ortu, A. Murgia, G. Destefanis, P. Tourani, R. Tonelli, M. Marchesi, and B. Adams, “The emotional side of software developers in JIRA,” in Proceedings of the 13th International Conference on Mining Software Repositories. ACM, 2016, pp. 480–483

    2016

  11. [11]

    SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text,

    M. R. Islam and M. F. Zibran, “SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text,” Journal of Systems and Software, vol. 145, pp. 125–146, 2018

    2018

  12. [12]

    Sentiment polarity detection for software development,

    F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli, “Sentiment polarity detection for software development,”Empirical Software En- gineering, vol. 23, no. 3, pp. 1352–1382, 2018

    2018

  13. [13]

    SentiCR: A customized sentiment analysis tool for code review interactions,

    T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi, “SentiCR: A customized sentiment analysis tool for code review interactions,” in2017 32nd IEEE/ACM International Conference on Automated Software Engineer- ing. IEEE, 2017, pp. 106–111

    2017

  14. [14]

    On negative results when using sentiment analysis tools for software engineering research,

    R. Jongeling, P. Sarkar, S. Datta, and A. Serebrenik, “On negative results when using sentiment analysis tools for software engineering research,” Empirical Software Engineering, vol. 22, pp. 2543–2584, 2017

    2017

  15. [15]

    Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended repli- cation study,

    N. Novielli, F. Calefato, F. Lanubile, and A. Serebrenik, “Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended repli- cation study,”Empirical Software Engineering, vol. 26, no. 4, p. 77, 2021

    2021

  16. [16]

    Sentiment analysis tools in software engineering: A systematic mapping study,

    M. Imran, A. Menon, M. Abubakar, W. Mahmood, and M. Niazi, “Sentiment analysis tools in software engineering: A systematic mapping study,”Information and Software Technology, vol. 151, p. 107018, 2022

    2022

  17. [17]

    “looks good to me ;-)

    D. Coutinho, L. Cito, M. V . Lima, B. Arantes, J. Alves Pereira, J. Arriel, J. Godinho, V . Martins, P. V . C. F. Lib´orio, L. Leite, A. Garcia, W. K. G. Assunc ¸˜ao, I. Steinmacher, A. Baffa, and B. Fonseca, ““looks good to me ;-)”: Assessing sentiment analysis tools for pull request discussions,” inProceedings of the 28th International Conference on Eva...

    2024

  18. [18]

    Impact of data quality for automatic issue classification using pre-trained language models,

    G. Colavito, F. Lanubile, N. Novielli, and L. Quaranta, “Impact of data quality for automatic issue classification using pre-trained language models,”Journal of Systems and Software, vol. 210, p. 111838, 2024

    2024

  19. [19]

    Detecting non-natural language artifacts for de- noising bug reports,

    T. Hirsch and B. Hofer, “Detecting non-natural language artifacts for de- noising bug reports,”Automated Software Engineering, vol. 29, no. 2, p. 52, 2022

    2022

  20. [20]

    Solving the problem of cas- cading errors: Approximate bayesian inference for linguistic annotation pipelines,

    J. R. Finkel, C. D. Manning, and A. Y . Ng, “Solving the problem of cas- cading errors: Approximate bayesian inference for linguistic annotation pipelines,” inProceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2006, pp. 618–626

    2006

  21. [21]

    When it’s all piling up: Inves- tigating error propagation in an NLP pipeline,

    T. Caselli, P. V ossen, M. van Erp, A. Fokkens, F. Ilievski, R. Izquierdo, M. Le, R. Morante, and M. Postma, “When it’s all piling up: Inves- tigating error propagation in an NLP pipeline,” inProceedings of the 20th International Conference on Applications of Natural Language to Information Systems. Springer, 2015, pp. 417–428

    2015

  22. [22]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out: Proceedings of the ACL-04 Workshop. Association for Computational Linguistics, 2004, pp. 74–81

    2004

  23. [23]

    Evaluating the factual consistency of abstractive text summarization,

    W. Kry ´sci´nski, B. McCann, C. Xiong, and R. Socher, “Evaluating the factual consistency of abstractive text summarization,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2020, pp. 9332– 9346

    2020

  24. [24]

    SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation,

    E. Clark, S. Rijhwani, S. Gehrmann, J. Maynez, R. Aharoni, V . Nikolaev, T. Sellam, A. Siddhant, D. Das, and A. Parikh, “SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023, pp. 9397–9413

    2023

  25. [25]

    GPTScore: Evaluate as You Desire

    J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “GPTScore: Evaluate as you desire,”arXiv preprint arXiv:2302.04166, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    SemEval 2017 task 10: ScienceIE – extracting keyphrases and relations from scientific publications,

    I. Augenstein, M. Das, S. Riedel, L. Vikraman, and A. McCallum, “SemEval 2017 task 10: ScienceIE – extracting keyphrases and relations from scientific publications,” inProceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 2017, pp. 546–555

    2017

  27. [27]

    What are developers talking about? an analysis of topics and trends in Stack Overflow,

    A. Barua, S. W. Thomas, and A. E. Hassan, “What are developers talking about? an analysis of topics and trends in Stack Overflow,”Empirical Software Engineering, vol. 19, no. 3, pp. 619–654, 2014

    2014

  28. [28]

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure

    M. Grootendorst, “BERTopic: Neural topic modeling with a class-based TF-IDF procedure,”arXiv preprint arXiv:2203.05794, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  29. [29]

    UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

    L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,”arXiv preprint arXiv:1802.03426, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  30. [30]

    Density-based clustering based on hierarchical density estimates,

    R. J. G. B. Campello, D. Moulavi, and J. Sander, “Density-based clustering based on hierarchical density estimates,” inAdvances in Knowledge Discovery and Data Mining. Springer, 2013, pp. 160–172

    2013

  31. [31]

    Density-based clustering validation,

    D. Moulavi, P. A. Jaskowiak, R. J. G. B. Campello, A. Zimek, and J. Sander, “Density-based clustering validation,” inProceedings of the 2014 SIAM International Conference on Data Mining. SIAM, 2014, pp. 839–847

    2014

  32. [32]

    A combination of opinion mining and social network techniques for discussion analysis,

    A. Stavrianou, J. Velcin, and J.-H. Chauchat, “A combination of opinion mining and social network techniques for discussion analysis,”Revue des Nouvelles Technologies de l’Information, vol. RNTI-E-17, pp. 25– 44, 2009

    2009

  33. [33]

    Leveraging large language models to identify conversation threads in collaborative learning,

    P. Ravi, D. W. Lee, B. Flamia, J. David, B. Hanks, C. Breazeal, E. Anderson, and G. Lin, “Leveraging large language models to identify conversation threads in collaborative learning,”arXiv preprint arXiv:2510.22844, 2025

    work page arXiv 2025

  34. [34]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering. IEEE, 2023, pp. 31–53

    2023

  35. [35]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474

    2020

  36. [36]

    Unveiling the potential of a conversational agent in developer support: Insights from mozilla’s PDF.js project,

    J. Correia, M. C. Nicholson, D. Coutinho, C. Barbosa, M. Castelluccio, M. Gerosa, A. Garcia, and I. Steinmacher, “Unveiling the potential of a conversational agent in developer support: Insights from mozilla’s PDF.js project,” inProceedings of the 1st ACM International Conference on AI- Powered Software. ACM, 2024, pp. 10–18

    2024

  37. [37]

    A comparison of conversational models and humans in answering technical questions: The Firefox case,

    J. Correia, D. Coutinho, M. Castelluccio, C. Barbosa, R. de Mello, A. Sarma, A. Garcia, M. Gerosa, and I. Steinmacher, “A comparison of conversational models and humans in answering technical questions: The Firefox case,”arXiv preprint arXiv:2510.21933, 2025

    work page arXiv 2025

  38. [38]

    V ADER: A parsimonious rule-based model for sentiment analysis of social media text,

    C. J. Hutto and E. Gilbert, “V ADER: A parsimonious rule-based model for sentiment analysis of social media text,” inProceedings of the International AAAI Conference on Web and Social Media, vol. 8, no. 1, 2014, pp. 216–225

    2014