{"id":"96d861a6-07c5-43fe-9407-6ab035a66561","arxiv_id":"2501.15922","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SkillScope predicts fine-grained skills for GitHub issues, but its reported 91% precision is measured against its own automatically generated labels.","lead":"SkillScope is a tool that predicts which of 217 fine-grained programming skills are needed to solve GitHub issues in Java projects, using Random Forest and large language models. The paper reports 91% precision, but the skill labels used for training and testing come from the same automated pipeline, so the accuracy claim lacks external validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 91/88/89 metrics are trained and evaluated against GPT-4o-mini-generated domain/subdomain labels; without human validation they measure self-consistency with the auto-labeler, not the skills actually needed.","rationale":"The reader's weakest assumption is exactly the load-bearing one. SkillScope's abstract and Section IV claim high precision, recall, and F-measure for predicting 217 fine-grained skills. Those skills are defined by labels produced by a zero-shot GPT-4o-mini pipeline in Section III-B, and no human validation is reported anywhere in the paper; in fact, Section V lists an empirical study with contributors as future work. The evaluation on JabRef appears to use the same automatic labeling pipeline as ground truth, so the metric is best interpreted as agreement with the auto-labeler, not with any externally verified notion of the skills needed to solve an issue. This concern is not about style or presentation: it undermines the central quantitative claim. The paper does have real strengths: a working tool, replication packages, and a reasonable RF-versus-LLM comparison, but those do not supply external validity. The appropriate verdict remains rejection unless the authors provide human-validated labels or an independent gold standard. The reader's analysis therefore stands without modification.","tokens_in":6674,"tokens_out":5136,"duration_ms":50635,"concrete_test":"Select a random sample of 200-300 issues/PRs from the JabRef evaluation set. Have two independent human annotators with Java/OSS experience label the 217 domains/subdomains, resolving disagreements by discussion or a third expert; report Cohen's kappa for inter-annotator agreement. Compare SkillScope's predictions against the human gold set using the same micro-averaged precision/recall/F1 used in Table IV. If the human-validated F1 is materially lower than 0.889 (e.g., below 0.80) or annotator agreement is poor, the headline claim is unsupported. Also re-run the evaluation on a sample where the GPT-4o-mini-generated labels have been manually corrected to measure label-noise impact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Section IV result that SkillScope RF + TF-IDF achieves 0.908 precision, 0.876 recall, and 0.889 F-measure on JabRef. The load-bearing premise is the correctness of the 217 labels. Section III-B assigns labels by taking class names and method names from ASTs and feeding them through zero-shot GPT-4o-mini prompts: first the domain, then the subdomain given the domain, with a Spacy similarity filter to select from the 217-label list. The paper reports no human validation of these labels, and Section V explicitly defers an empirical study to future work. The Table IV evaluation is performed on JabRef contributions processed through the same tool pipeline, so the automatically generated labels appear to serve as ground truth for both training and testing. If GPT-4o-mini's zero-shot mappings are noisy or systematically biased, the RF model can learn to reproduce those biases and still obtain high micro-F1 against the same auto-labeler. The reported numbers then quantify agreement with the labeling pipeline rather than the true skills needed to solve issues. This is a circularity risk, not a minor annotation-accuracy issue, because the granularity claim depends precisely on the 186 machine-added subdomains that have never been checked by humans.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SkillScope, a tool that predicts fine-grained, multilevel API-domain and subdomain labels (217 total) for GitHub issues in Java projects. The tool mines closed pull requests and issue text, parses Java source files into ASTs, extracts class and method names, uses zero-shot GPT-4o-mini prompts with a Spacy similarity filter to assign domain/subdomain labels, and then trains Random Forest and fine-tuned GPT models on one-hot encoded labels. The evaluation on the JabRef project reports precision 0.908, recall 0.876, and F-measure 0.889 for SkillScope RF+TF-IDF, which the paper claims surpasses prior 31-domain approaches. A user survey with eight respondents is also reported.","tokens_in":6963,"tokens_out":2733,"duration_ms":25631,"significance":"If the reported accuracy were valid, SkillScope would be a useful contribution to newcomer onboarding in OSS, extending prior flat 31-domain skill labels to a 217-label two-level hierarchy. Strengths of the paper include a fully automated pipeline, a publicly available tool and replication packages, and a direct comparison with several prior methods on the same project. However, the central evaluation claim is undermined by the fact that the ground-truth labels used for both training and testing are generated by the same automated GPT-4o-mini pipeline that the tool itself uses, with no human gold standard. The reported metrics therefore primarily measure self-consistency with the label-generation process, not the actual skills needed to solve issues. Since the paper explicitly defers an empirical study with contributors to future work, the central claim of accurate skill prediction is not supported by the evidence presented.","major_comments":[{"comment":"The ground-truth labels for both training and evaluation are produced by the same automated pipeline that the tool uses for prediction. Section III-B describes how class and method names extracted via ASTs are sent through zero-shot GPT-4o-mini prompts, with a Spacy similarity filter selecting from the 217 labels. Section IV evaluates the tool on JabRef contributions processed through the same tool pipeline, and the automatically generated labels appear to serve as ground truth. Consequently, the precision, recall, and F-measure reported in Table IV measure agreement between the Random Forest predictions and the auto-labeler's outputs, not the actual skills needed to resolve issues. The paper reports no human validation of the 217 labels; Section V explicitly defers an empirical study with contributors to future work. This circularity is load-bearing because the central claim of the paper is the numerical accuracy of the predictions.","section":"Section III-B, III-C, IV"},{"comment":"The comparison with prior work is not apples-to-apples. Section II states that Vargovich et al.'s approach involves expert review and manual categorization of APIs, and Section V states that Santos et al. evaluated their API domains with students and practitioners from industry. In contrast, SkillScope's labels are generated without any human validation. If the prior methods were evaluated against human-validated labels while SkillScope is evaluated against its own auto-generated labels, then the claim that SkillScope 'overcame the similar tools with a precision of 91%' (Section I) is not supported by the table. A shared gold standard, or at least a human-annotated subset, is needed to make the comparison meaningful.","section":"Section IV, Table IV"},{"comment":"The user survey has only eight respondents, and the paper provides no details about the recruitment, the survey instrument, or how respondents interacted with the tool. The statement that 'the majority finding that the tool either strongly or somewhat confident to contribute' is based on a very small, self-selected sample and, as reported, cannot support a general claim about user experience. Additionally, no statistical tests or confidence intervals are reported for the main precision/recall/F-measure results, so the observed differences between models (e.g., 0.908 vs. 0.864 for RF+TF-IDF) are not shown to be statistically significant.","section":"Section IV, Fig. 2"}],"minor_comments":[{"comment":"The mapping that 'the class name corresponds to the API domain, while the function name corresponds to the subdomain' is a strong simplification that is likely to introduce noise, since many method names are not informative about skills (e.g., 'toString', 'get', 'set'). This assumption should be acknowledged and ideally checked against human intuition.","section":"Section III-B"},{"comment":"There is an inconsistency in the model names: the text first says 'fine-tune a GPT-4o model' and later says 'Both models are fine-tuned GPT-4o-mini models.' Please clarify which model was actually used.","section":"Section III-C"},{"comment":"The table footnote '*all classes' is ambiguous: it is unclear whether this refers to micro-averaging over all classes or to a specific subset. Please state explicitly how the metrics are averaged (the text says micro-averaging, but the table caption and footnotes should be consistent).","section":"Section IV, Table IV"},{"comment":"The name 'SkilScope' is a typo for 'SkillScope'.","section":"Section V"},{"comment":"The phrase 'Collavito et al. [7] investigated the extension to its LLMs' is grammatically awkward and should be rewritten for clarity.","section":"Section II"}],"recommendation":"reject","confidential_remarks":"The circularity in the evaluation is the primary reason for rejection: the reported metrics compare model outputs to labels generated by the same GPT-4o-mini pipeline used to create training data, with no human gold standard. This is not a fixable presentation issue; the paper explicitly defers the required empirical validation to future work. If the authors reframed the contribution as a tool demonstration without the accuracy claims, or added a human-validated evaluation, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid tool paper with a load-bearing evaluation flaw. The authors extend Santos et al.'s 31 API domains to 217 hierarchical labels (186 subdomains generated by GPT-4o-mini), build a full pipeline from GitHub mining through AST parsing to predictions, and ship a UI and replication packages. That's real work, and the tool would be useful if the labels were trustworthy.\n\nThe problem is the evaluation. The ground-truth labels for both training and testing come from the same zero-shot GPT-4o-mini pipeline that produces the subdomain taxonomy (Section III-B). The JabRef test set in Section IV is processed through that same pipeline, so the RF's 0.908 precision / 0.876 recall / 0.889 F-measure measures agreement with the auto-labeler, not the skills actually needed to solve issues. The paper acknowledges that an empirical study with contributors is future work (Section V), which is honest but does not support the abstract's claim. The comparison to Santos et al. is also tricky: if those baselines used human-validated labels, then you're comparing a self-consistency number to a human-validated number.\n\nWhat's good: the multi-level taxonomy is a reasonable idea, the MLSMOTE and TF-IDF setup follows prior work, the RF vs GPT-3.5 vs GPT-4o-mini comparison is useful (even if LLMs lose), and the replication packages are a plus. The survey is too small (n=8) to say much, but that's a minor point.\n\nThe circularity is the main issue, and it is fixable. A human-annotated sample of a few hundred issues, or even validating a sample of the auto-generated labels, would let the authors report a decontaminated number. Without that, the headline claim is not supported.\n\nWho is this for: researchers working on issue labeling and OSS onboarding. The paper deserves a serious referee because the tool and dataset address a real problem and the core flaw is correctable with additional validation. I'd set the desk decision to major revision, not reject.","headline":"Useful engineering extension to issue-skill labeling, but the 91/88/89 numbers measure self-consistency with the auto-labeler, not validated skill prediction.","tokens_in":7487,"tokens_out":2331,"would_cite":true,"duration_ms":22022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillScope introduces a fully automatic tool that predicts 217 fine-grained, two-level API-domain skills for open issues, with the random-forest classifier reaching 91% precision, 88% recall, and 89% F-measure and beating fine-tuned LLMs.","keywords":["software engineering","skill categorization","open source software","machine learning","large language models","issue labeling","API domains","random forest"],"falsifier":"Take a random sample of, say, 100 issues labeled by SkillScope, have human annotators familiar with the codebases judge whether each predicted domain and subdomain is genuinely required to resolve the issue, and measure the agreement; if agreement falls substantially below the reported 91% precision, the reported metrics do not reflect real-world accuracy.","tokens_in":6506,"feed_emoji":"🛠️","tokens_out":13820,"duration_ms":98294,"temperature":0.7,"pith_summary":"SkillScope is a fully automatic tool that labels open issues with the fine-grained programming skills needed to resolve them. It builds a two-level taxonomy of 217 API-domain and subdomain labels from Java source code, then trains classifiers to predict which labels apply to each issue's text. In a case study on a Java project, the random-forest model with TF-IDF features achieved 91% precision, 88% recall, and 89% F-measure, outperforming prior flat-label approaches and fine-tuned GPT models. If these figures hold, the tool could help new contributors find tasks that match their specific expertise and help maintainers delegate work more effectively.","feed_headline":"Predicts fine-grained coding skills for open issues at 91% precision","feed_subtitle":"Fine-grained skill labels could help newcomers find tasks they can actually solve.","key_machinery":"The key machinery is the two-level API-domain taxonomy and the automated pipeline that derives it: the paper treats class names as API domains and method names as subdomains, parses Java source files into abstract syntax trees, and uses zero-shot prompting of a small LLM to assign each class and method to a domain and then a subdomain, with a similarity function constraining outputs to the known label set. The resulting 217 binary labels are used to train per-domain classifiers—random forest with TF-IDF and fine-tuned GPT-4o-mini—in a one-vs-all scheme, after MLSMOTE oversampling for the random forest and synthetic data generation for the LLM.","core_discovery":"The central discovery is that a relatively simple pipeline—parse Java source files into abstract syntax trees, use zero-shot GPT-4o-mini to map class and method names to a hierarchy of 31 API domains and 186 subdomains, then train one-vs-all binary classifiers per label—can predict the multilevel skills required by an issue. In a case study on a Java project, the random-forest + TF-IDF variant scored 0.908 precision, 0.876 recall, and 0.889 F-1, exceeding all previously published issue-labeling tools. The fine-tuned GPT-4o-mini and GPT-3.5 models lagged behind, showing that the LLM advantage is not automatic for this fine-grained, long-tailed classification task.","pith_inferences":["In our reading, the reported metrics measure agreement with an LLM-generated label set rather than with human judgment, so the real-world precision could be lower; a human-annotated test set would be needed to learn the true gap.","The result that a classical model beats a fine-tuned LLM on this highly imbalanced, long-tailed classification task suggests that synthetic data augmentation and fine-tuning budgets may not be sufficient to overcome label noise or data scarcity in code-focused tasks.","The 217-label taxonomy built from class and method structure could serve as a reusable benchmark for issue labeling beyond this tool, independent of the prediction models.","A natural next step, not tested here, would be to use the predicted subdomain labels to match issues to contributors by mining their contribution history; the paper leaves this for future work."],"forward_implications":["If the reported accuracy generalizes beyond the case study, issue trackers could automatically expose the precise skills each task demands, lowering the barrier for newcomers.","The two-level labels give contributors more context than flat domains, such as seeing a database subdomain like query execution instead of just 'Database', which could reduce mismatches between contributor skills and task demands.","The finding that random forest with TF-IDF outperforms fine-tuned LLMs suggests lightweight classifiers remain competitive for issue labeling, with lower computational cost and faster predictions.","The pipeline is limited to Java, but the same class-as-domain, method-as-subdomain idea could be adapted to other languages by replacing the parser."],"supporting_citations":[{"why":"Provides a related method for matching issues to skills and a baseline result in the comparison table.","marker":"[3]"},{"why":"Defines the 31 API-domain labels and the random-forest plus TF-IDF baseline that SkillScope extends.","marker":"[4]"},{"why":"Supplies a tool baseline using BERT and random forest with the same flat domains.","marker":"[5]"},{"why":"Establishes the earlier API-based skill prediction approach and the evaluation metrics used throughout.","marker":"[6]"},{"why":"Offers the LLM-based issue labeling baseline against which the GPT models are measured.","marker":"[7]"},{"why":"Provides the fine-tuning recipe for LLM issue classification that SkillScope adapts for its small models.","marker":"[8]"},{"why":"Supports the choice of TF-IDF vectorization over word embeddings for small, unstructured GitHub issue text.","marker":"[22]"}],"fun_headline_variants":["SkillScope predicts 186 subdomain skills for GitHub issues","91% precision: predicting skills needed for Java issues","Fine-grained skill prediction for open source issues","SkillScope predicts multilevel skills for GitHub issues at 91%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that the LLM-generated domain and subdomain labels assigned to classes and methods are accurate enough to serve as ground truth for both training and testing; the paper does not report any human validation of those labels.","fun_headline_variants_meta":{"raw":{"variants":["SkillScope predicts 186 subdomain skills for GitHub issues","91% precision: predicting skills needed for Java issues","Fine-grained skill prediction for open source issues","SkillScope predicts multilevel skills for GitHub issues at 91%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2859,"prompt_tokens":909,"completion_tokens":1950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1885}},"tokens_in":525,"tokens_out":1950,"duration_ms":13335,"temperature":1.0,"reasoning_tokens":1885,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:49:20.181591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 100 issues labeled by SkillScope, have human annotators familiar with the codebases judge whether each predicted domain and subdomain is genuinely required to resolve the issue, and measure the agreement; if agreement falls substantially below the reported 91% precision, the reported metrics do not reflect real-world accuracy.","supporting_citations":[{"cited_title":"Tell me who are you talking to and i will tell you what issues need your skills,","cited_arxiv_id":null,"evidence_quote":"Provides a related method for matching issues to skills and a baseline result in the comparison table."},{"cited_title":"Tag that issue: Applying api-domain labels in issue tracking systems,","cited_arxiv_id":null,"evidence_quote":"Defines the 31 API-domain labels and the random-forest plus TF-IDF baseline that SkillScope extends."},{"cited_title":"Givemelabeledissues: An open source issue recommen- dation system,","cited_arxiv_id":null,"evidence_quote":"Supplies a tool baseline using BERT and random forest with the same flat domains."},{"cited_title":"Can i solve it? identifying apis required to complete oss tasks,","cited_arxiv_id":null,"evidence_quote":"Establishes the earlier API-based skill prediction approach and the evaluation metrics used throughout."},{"cited_title":"Applying large language models to issue classification,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuning recipe for LLM issue classification that SkillScope adapts for its small models."},{"cited_title":"Performance comparison of tf-idf and word2vec models for emotion text classification,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of TF-IDF vectorization over word embeddings for small, unstructured GitHub issue text."}],"review_version":1}