pith. sign in

arxiv: 2606.08096 · v1 · pith:NDHIZ2TZnew · submitted 2026-06-06 · 💻 cs.SE

Identifying unique developers in OSS projects: A family of models

Ruoyu Su , Alexander Bakhtin , Matteo Esposito , Davide Taibi , Valentina Lenarduzzi This is my paper

Pith reviewed 2026-06-27 19:34 UTC · model grok-4.3

classification 💻 cs.SE
keywords OSS developer de-duplicationalias detectionmachine learningcommit metadataground truth datasetprecision-cost trade-offLLM-assisted labelingsoftware mining
0
0 comments X

The pith

LLM-assisted dataset enables benchmarking of ML models for OSS developer de-duplication balancing precision and computational cost

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Commit metadata in open source projects often lists the same developer under multiple names and emails, which can distort metrics like organizational coupling. This paper creates a large ground-truth dataset of duplicate identities by having an LLM propose matches and then manually validating them. It uses this dataset to train and test several classical machine learning models of different levels of complexity. The evaluation tracks not only how accurate each model is at identifying duplicates but also the time and energy needed to train and run them. The result is guidance on selecting a model that fits the accuracy needs and compute budget of large-scale mining tasks.

Core claim

We construct a dataset of duplicate developer identities using an LLM-assisted matching process with manual validation. We train a family of classical ML models on this dataset and evaluate them against a baseline of Indel similarity, focusing on precision as well as the computational cost in terms of training time, inference time, and energy consumption. This produces a benchmark that identifies which models provide the best trade-off for scalable OSS developer identification.

What carries the argument

The LLM-assisted matching process with manual validation that creates the ground-truth dataset of duplicate identities for training and comparing the ML models.

If this is right

  • Correct identification of unique developers improves the reliability of coupling metrics used in OSS analysis.
  • Lower-complexity ML models can achieve high precision while requiring less time and energy for training and inference.
  • Practitioners gain concrete criteria for choosing a de-duplication method based on project scale and available resources.
  • The created dataset can serve as a reference for developing or testing new de-duplication techniques.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Applying these de-duplication models could standardize developer tracking across different OSS platforms and tools.
  • Similar methods might address identity resolution in other domains with noisy metadata, such as academic publications or social media.
  • Further scaling the dataset to include more languages or project types could test the generalizability of the benchmark results.

Load-bearing premise

The LLM-assisted matching process combined with manual validation produces a sufficiently accurate and unbiased ground-truth dataset of duplicate identities.

What would settle it

An independent manual review of a random sample from the dataset showing a high rate of incorrect duplicate labels, or the top-performing model failing to generalize to an independently collected set of developer identities.

read the original abstract

Organizational and logical coupling metrics require reliable identification of unique developers. In OSS, commit metadata is limited to names and emails, and the same developer may appear under multiple aliases, which can distort coupling measurements if de-duplication is missing. We aim to build a scalable and accurate pipeline for OSS developer de-duplication and to provide guidance on choosing a model based on precision vs. computational effort. We use Indel similarity as a baseline, then run an LLM-assisted matching process with manual validation to create a large dataset of duplicate identities. Using this dataset, we train and compare classical ML models of different complexity, evaluating precision along with training and inference time and energy. We expect a high-quality dataset and a benchmark of approaches that clarifies which solutions offer the best trade-off between accuracy and cost for large-scale OSS mining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents a pipeline for de-duplicating developer identities in OSS projects using commit metadata. It uses Indel similarity as a baseline, employs an LLM-assisted process with manual validation to generate a ground-truth dataset of duplicate identities, and then trains and benchmarks classical ML models of varying complexity on this dataset. The goal is to provide a high-quality dataset and guidance on selecting models based on the trade-off between precision and computational cost (training/inference time and energy) for large-scale OSS mining.

Significance. If the ground-truth dataset proves accurate and unbiased, the work would deliver a useful community resource and empirical benchmark for developer de-duplication methods, which is important for improving the reliability of organizational and logical coupling metrics in software engineering research.

major comments (2)
  1. [Dataset Construction] Dataset Construction section: The LLM-assisted matching process combined with manual validation is the foundational step for all quantitative results. The manuscript should provide more details on the specific LLM used, prompt templates, criteria for manual validation, and any measures taken to assess label accuracy (e.g., agreement rates or comparison to external sources) to address potential biases in name/email formatting common in OSS data.
  2. [Evaluation and Results] Evaluation and Results section: The abstract states aims and high-level method but supplies no quantitative results, dataset sizes, precision numbers, or error analysis. The full manuscript must report these metrics (including validation of LLM-generated labels) to support the claimed accuracy-vs-cost trade-offs; their absence prevents assessment of the central benchmarking claims.
minor comments (2)
  1. [Introduction] Introduction: The term 'Indel similarity' is used as baseline without definition or citation; add a brief explanation or reference.
  2. [Abstract] Abstract: Consider adding one or two key quantitative findings on model performance or dataset scale if space permits.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the valuable feedback on our manuscript. The comments highlight important areas for improving the description of our dataset construction process and the reporting of quantitative results. We address each point below and will revise the manuscript accordingly to enhance clarity and completeness.

read point-by-point responses

  1. Referee: [Dataset Construction] Dataset Construction section: The LLM-assisted matching process combined with manual validation is the foundational step for all quantitative results. The manuscript should provide more details on the specific LLM used, prompt templates, criteria for manual validation, and any measures taken to assess label accuracy (e.g., agreement rates or comparison to external sources) to address potential biases in name/email formatting common in OSS data.

    Authors: We agree that additional details on the LLM-assisted process are essential for reproducibility. In the revised version, we will include the specific LLM (including version), the prompt templates used, the exact criteria for manual validation, and measures of label accuracy such as agreement rates between validators. This will also help mitigate concerns about biases in OSS name and email data. revision: yes

  2. Referee: [Evaluation and Results] Evaluation and Results section: The abstract states aims and high-level method but supplies no quantitative results, dataset sizes, precision numbers, or error analysis. The full manuscript must report these metrics (including validation of LLM-generated labels) to support the claimed accuracy-vs-cost trade-offs; their absence prevents assessment of the central benchmarking claims.

    Authors: The abstract is intentionally high-level, but we will ensure the full manuscript's Evaluation and Results section reports all necessary quantitative details, including dataset sizes, precision/recall metrics, error analysis, and validation of the LLM labels. These will be expanded if not already sufficiently present to fully substantiate the precision vs. computational cost trade-offs. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmarking with independent ground-truth construction

full rationale

The paper describes an empirical pipeline: Indel baseline, LLM-assisted matching with manual validation to build a duplicate-identity dataset, then training/comparing classical ML models on that dataset while measuring precision, time, and energy. No equations, fitted parameters presented as predictions, self-definitional loops, or load-bearing self-citations appear in the provided text. The central deliverable (dataset + benchmark) rests on the external validity of the LLM+manual process rather than any derivation that reduces to its own inputs by construction. This is a standard empirical study whose claims are falsifiable against held-out data or alternative ground-truth methods.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only abstract available; ledger populated from stated elements only.

axioms (1)
  • domain assumption Indel similarity provides a meaningful baseline for developer identity matching
    Explicitly used as baseline in the abstract.

pith-pipeline@v0.9.1-grok · 5671 in / 1124 out tokens · 17056 ms · 2026-06-27T19:34:24.683099+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

141 extracted references · 2 canonical work pages

  1. [1]

    A systematic mapping study , author =

    Metrics and models for developer collaboration analysis in microservice-based systems. A systematic mapping study , author =. CEUR Workshop Proceedings , publisher =

  2. [2]

    2016 IEEE 23rd international conference on software analysis, evolution, and Reengineering (SANER) , volume=

    Forking and the Sustainability of the Developer Community Participation--An Empirical Investigation on Outcomes and Reasons , author=. 2016 IEEE 23rd international conference on software analysis, evolution, and Reengineering (SANER) , volume=

    2016

  3. [3]

    Doklady Akademii Nauk , volume=

    Binary codes capable of correcting deletions, insertions, and reversals , author=. Doklady Akademii Nauk , volume=. 1965 , organization=

    1965

  4. [4]

    Biometrics , pages=

    Sample size for case-control studies using Cochran's statistic , author=. Biometrics , pages=. 1986 , publisher=

    1986

  5. [5]

    6th India Software Engineering Conference , pages=

    Evolution of developer social network and its impact on bug fixing process , author=. 6th India Software Engineering Conference , pages=

  6. [6]

    30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages=

    Quantifying community evolution in developer social networks , author=. 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages=

  7. [7]

    2013 17th European conference on software maintenance and reengineering , pages=

    Network structure of social coding in github , author=. 2013 17th European conference on software maintenance and reengineering , pages=

    2013

  8. [8]

    ACM Transactions on Software Engineering and Methodology , volume=

    Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based Indices , author=. ACM Transactions on Software Engineering and Methodology , volume=. 2024 , publisher=

    2024

  9. [9]

    international AAAI conference on web and social media , volume=

    Coding together at scale: GitHub as a collaborative social network , author=. international AAAI conference on web and social media , volume=

  10. [10]

    2008 international workshop on Cooperative and human aspects of software engineering , pages=

    What dynamic network metrics can tell us about developer roles , author=. 2008 international workshop on Cooperative and human aspects of software engineering , pages=

    2008

  11. [11]

    2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) , pages=

    Classifying developers into core and peripheral: An empirical study on count and network metrics , author=. 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) , pages=

    2017

  12. [12]

    Decision Support Systems , volume=

    Reputation management in an open source developer social network: An empirical study on determinants of positive evaluations , author=. Decision Support Systems , volume=. 2012 , publisher=

    2012

  13. [13]

    2016 23rd Asia-Pacific Software Engineering Conference (APSEC) , pages=

    Task recommendation with developer social network in software crowdsourcing , author=. 2016 23rd Asia-Pacific Software Engineering Conference (APSEC) , pages=

    2016

  14. [14]

    2024 International Conference on Big Data Analytics in Bioinformatics (DABCon) , pages=

    Exploring Fusion Centrality in Developer Social Networks , author=. 2024 International Conference on Big Data Analytics in Bioinformatics (DABCon) , pages=

    2024

  15. [15]

    International Journal of Human-Computer Studies , volume=

    User and developer mediation in an Open Source Software community: Boundary spanning through cross participation in online discussions , author=. International Journal of Human-Computer Studies , volume=. 2008 , publisher=

    2008

  16. [16]

    2013 International conference on social computing , pages=

    Stackoverflow and github: Associations between software development and crowdsourced knowledge , author=. 2013 International conference on social computing , pages=

    2013

  17. [17]

    2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages=

    Studying software developer expertise and contributions in Stack Overflow and GitHub , author=. 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pages=

    2020

  18. [18]

    World Wide Web , volume=

    Scsminer: mining social coding sites for software developer recommendation with relevance propagation , author=. World Wide Web , volume=. 2018 , publisher=

    2018

  19. [19]

    Journal of Management Information Systems , volume=

    Developer heterogeneity and formation of communication networks in open source software projects , author=. Journal of Management Information Systems , volume=. 2010 , publisher=

    2010

  20. [20]

    Collaboration and Technology: 18th International Conference, CRIWG 2012 Raesfeld, Germany, September 16-19, 2012 Proceedings 18 , pages=

    Characterizing key developers: a case study with apache ant , author=. Collaboration and Technology: 18th International Conference, CRIWG 2012 Raesfeld, Germany, September 16-19, 2012 Proceedings 18 , pages=. 2012 , organization=

    2012

  21. [21]

    European Conference on Service-Oriented and Cloud Computing , pages=

    One microservice per developer: is this the trend in OSS? , author=. European Conference on Service-Oriented and Cloud Computing , pages=. 2023 , organization=

    2023

  22. [22]

    46th International Conference on Software Engineering: Software Engineering in Practice , pages=

    OpenRank leaderboard: motivating open source collaborations through social network evaluation in Alibaba , author=. 46th International Conference on Software Engineering: Software Engineering in Practice , pages=

  23. [23]

    2006 international workshop on Mining software repositories , pages=

    Mining email social networks , author=. 2006 international workshop on Mining software repositories , pages=

    2006

  24. [24]

    2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE) , pages=

    Characterizing and understanding software developer networks in security development , author=. 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE) , pages=

    2021

  25. [25]

    2nd international workshop on Evidential assessment of software technologies , pages=

    An empirical study on identifying core developers using network analysis , author=. 2nd international workshop on Evidential assessment of software technologies , pages=

  26. [26]

    2012 28th IEEE International Conference on Software Maintenance (ICSM) , pages=

    Who's who in Gnome: Using LSA to merge software repository identities , author=. 2012 28th IEEE International Conference on Software Maintenance (ICSM) , pages=

    2012

  27. [27]

    Involvement, contribution and influence in github and stack overflow , author=

  28. [28]

    Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Proceedings, Part II 9 , pages=

    Github and stack overflow: Analyzing developer interests across multiple social collaborative platforms , author=. Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Proceedings, Part II 9 , pages=. 2017 , organization=

    2017

  29. [29]

    ACM SIGSOFT Software Engineering Notes , volume=

    Developer identification methods for integrated data from various sources , author=. ACM SIGSOFT Software Engineering Notes , volume=. 2005 , publisher=

    2005

  30. [30]

    Sixth IEEE International Conference on Data Mining-Workshops (ICDMW'06) , pages=

    A comparison of personal name matching: Techniques and practical issues , author=. Sixth IEEE International Conference on Data Mining-Workshops (ICDMW'06) , pages=

  31. [31]

    17th International Conference on Mining Software Repositories , pages=

    A mixed graph-relational dataset of socio-technical interactions in open source systems , author=. 17th International Conference on Mining Software Repositories , pages=

  32. [32]

    International Conference on Software and System Processes (ICSSP) , pages=

    Do communities in developer interaction networks align with subsystem developer teams? An empirical study of open source systems , author=. International Conference on Software and System Processes (ICSSP) , pages=

  33. [33]

    Science of Computer Programming , volume=

    A comparison of identity merge algorithms for software repositories , author=. Science of Computer Programming , volume=. 2013 , publisher=

    2013

  34. [34]

    ACM Transactions on Software Engineering and Methodology , volume=

    Automatic core-developer identification on GitHub: A validation study , author=. ACM Transactions on Software Engineering and Methodology , volume=. 2023 , publisher=

    2023

  35. [35]

    41st Annual Hawaii International Conference on System Sciences (HICSS 2008) , pages=

    An exploratory study on the evolution of oss developer communities , author=. 41st Annual Hawaii International Conference on System Sciences (HICSS 2008) , pages=

    2008

  36. [36]

    Computing , volume=

    Exploring microservice ownership and organizational coupling in open-source projects: an empirical study , author=. Computing , volume=. 2025 , publisher=

    2025

  37. [37]

    38th Annual Hawaii International Conference on System Sciences , pages=

    A topological analysis of the open souce software development community , author=. 38th Annual Hawaii International Conference on System Sciences , pages=

  38. [38]

    2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C) , pages=

    Toward Organizational Decoupling in Microservices Through Key Developer Allocation , author=. 2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C) , pages=

    2025

  39. [39]

    2010 43rd Hawaii International Conference on System Sciences , pages=

    The importance of social network structure in the open source software developer community , author=. 2010 43rd Hawaii International Conference on System Sciences , pages=

    2010

  40. [40]

    IEEE Access , volume=

    Empirical study on the evolution of developer social networks , author=. IEEE Access , volume=. 2018 , publisher=

    2018

  41. [41]

    2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement , volume =

    DevNet: Exploring Developer Collaboration in Heterogeneous Networks of Bug Repositories , author =. 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement , volume =

    2013

  42. [42]

    1st International Workshop on New Trends in Software Architecture , series =

    Understanding the Causes of Microservice Logical Coupling: an Exploratory Study , author =. 1st International Workshop on New Trends in Software Architecture , series =

  43. [43]

    2011 27th IEEE International Conference on Software Maintenance (ICSM) , volume =

    Understanding a developer social network and its evolution , author =. 2011 27th IEEE International Conference on Software Maintenance (ICSM) , volume =

    2011

  44. [44]

    Journal of Systems and Software , volume = 118, pages =

    Effect of developer collaboration activity on software quality in two large scale projects , author =. Journal of Systems and Software , volume = 118, pages =

  45. [45]

    ACM Trans

    The small-world effect: The influence of macro-level properties of developer collaboration networks on open-source project success , author =. ACM Trans. Softw. Eng. Methodol. , publisher =

  46. [46]

    International Journal of Data Mining & Knowledge Management Process (IJDKP) , volume =

    Mohammed Abufouda and Hadil Abukwaik , title =. International Journal of Data Mining & Knowledge Management Process (IJDKP) , volume =. 2017 , month =

    2017

  47. [47]

    2009 IEEE 31st International Conference on Software Engineering , volume =

    Predicting build failures using social network analysis on developer communication , author =. 2009 IEEE 31st International Conference on Software Engineering , volume =

    2009

  48. [48]

    16th ACM SIGSOFT International Symposium on Foundations of Software Engineering , pages =

    Predicting failures with developer networks and social network analysis , author =. 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering , pages =

  49. [49]

    2023 IEEE International Conference on Service-Oriented System Engineering (SOSE) , pages =

    Analyzing organizational structure of microservice projects based on contributor collaboration , author =. 2023 IEEE International Conference on Service-Oriented System Engineering (SOSE) , pages =

    2023

  50. [50]

    4th International Workshop on Cooperative and Human Aspects of Software Engineering , pages =

    Mining and visualizing developer networks from version control systems , author =. 4th International Workshop on Cooperative and Human Aspects of Software Engineering , pages =

  51. [51]

    International Conference on Product-Focused Software Process Improvement , pages =

    Evaluating Microservice Organizational Coupling Based on Cross-Service Contribution , author =. International Conference on Product-Focused Software Process Improvement , pages =

  52. [52]

    European Conference on Software Architecture , pages =

    Temporal Community Detection in Developer Collaboration Networks of Microservice Projects , author =. European Conference on Software Architecture , pages =

  53. [53]

    2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C) , pages =

    Microservice logical coupling: A preliminary validation , author =. 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C) , pages =

    2023

  54. [54]

    2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C) , volume =

    Toward Collaboration Optimization in Microservice Projects Based on Developer Personalities , author =. 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C) , volume =

    2024

  55. [55]

    MSR , volume =

    microSecEnD: A Dataset of Security-Enriched Dataflow Diagrams for Microservice Applications , author =. MSR , volume =

  56. [56]

    Tools Reconstructing Microservice Architecture: A Systematic Mapping Study , author =

  57. [57]

    JSS , issn =

    Automatic extraction of security-rich dataflow diagrams for microservice applications written in Java , author =. JSS , issn =

  58. [58]

    IST , issn =

    Guidelines for including grey literature and conducting multivocal literature reviews in software engineering , author =. IST , issn =

  59. [59]

    Symposium on Applied Computing , pages

    Attack graph generation for microservice architecture , author =. Symposium on Applied Computing , pages. =

  60. [60]

    Inforte Summer School on Software Maintenance and Evolution , publisher =

    A curated Dataset of Microservices-Based Systems , author =. Inforte Summer School on Software Maintenance and Evolution , publisher =

  61. [61]

    Advances in Service-Oriented and Cloud Computing , publisher =

    Mining the Architecture of Microservice-Based Applications from their Kubernetes Deployment , author =. Advances in Service-Oriented and Cloud Computing , publisher =

  62. [62]

    Enterprise, Business-Process and Information Systems Modeling , publisher =

    A Modeling Method for Systematic Architecture Reconstruction of Microservice-Based Software Systems , author =. Enterprise, Business-Process and Information Systems Modeling , publisher =

  63. [63]

    ASE , volume =

    Using Static Analysis to Address Microservice Architecture Reconstruction , author =. ASE , volume =

  64. [64]

    PeerJ Computer Science , volume = 7, month

    On automated RBAC assessment by constructing a centralized perspective for microservice mesh , author =. PeerJ Computer Science , volume = 7, month. =

  65. [65]

    EASE , publisher =

    Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering , author =. EASE , publisher =

  66. [66]

    EMSE , publisher =

    Benchmarking Kappa: Interrater Agreement in Software ProcessAssessments , author =. EMSE , publisher =

  67. [67]

    IEEE Access , number =

    Visualizing Microservice Architecture in the Dynamic Perspective: A Systematic Mapping Study , author =. IEEE Access , number =

  68. [68]

    JSS , volume

    The pains and gains of microservices: A Systematic grey literature review , author =. JSS , volume. = 146, pages. =

  69. [69]

    Design principles, architectural smells and refactorings for microservices: a multivocal review , author =

  70. [70]

    JSS , issn =

    Smells and refactorings for microservices security: A multivocal literature review , author =. JSS , issn =

  71. [71]

    Applied Sciences , publisher =

    Roadmap to Reasoning in Microservice Systems: A Rapid Review , author =. Applied Sciences , publisher =

  72. [72]

    Applied Sciences , publisher =

    On Microservice Analysis and Architecture Evolution: A Systematic Mapping Study , author =. Applied Sciences , publisher =

  73. [73]

    SECURWARE , volume =

    Comparison of Static Code Analysis Tools , author =. SECURWARE , volume =

  74. [74]

    Electronic Notes in Theoretical Computer Science , issn =

    A Comparative Study of Industrial Static Analysis Tools , author =. Electronic Notes in Theoretical Computer Science , issn =

  75. [75]

    JSS , issn =

    A critical comparison on six static analysis tools: Detection, agreement, and precision , author =. JSS , issn =

  76. [76]

    IJSSMET , publisher =

    A Comparative Study of Meta-Data-Based Microservice Extraction Tools , author =. IJSSMET , publisher =

  77. [77]

    JSS , issn =

    Monitoring tools for DevOps and microservices: A systematic grey literature review , author =. JSS , issn =

  78. [78]

    SCC , volume =

    Survey on Tools and Techniques Detecting Microservice API Patterns , author =. SCC , volume =

  79. [79]

    Microservices: a definition of this new architectural term , author =

  80. [80]

    Present and Ulterior Software Engineering , publisher =

    Microservices: yesterday, today, and tomorrow , author =. Present and Ulterior Software Engineering , publisher =

Showing first 80 references.