Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach

Amir Saberhabibi; Bedir Tekinerdogan; Behbod Eskafi; Michel Chaudron; Niloofar Nikfarjam; Sadegh Eskandari; Siamak Farshidi; Slinger Jansen

REVIEW 4 major objections 2 minor 1 cited by

Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach

T0 review · 4 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that Python package selection can be formalized as a multi-criteria decision problem, and that a data-backed system called PySelect beats generic generative-AI tools in recommendation quality.

desk verdict PySelect looks like a sensible integration of existing techniques, but the abstract's main claim about recommendation quality is unverifiable until the evaluation ground truth and baselines are specified. read the letter →

arxiv 2508.05693 v1 pith:7JQEEUIP submitted 2025-08-06 cs.SE cs.AI

Siamak Farshidi , Amir Saberhabibi , Behbod Eskafi , Niloofar Nikfarjam , Sadegh Eskandari , Slinger Jansen , Michel Chaudron , Bedir Tekinerdogan This is my paper · ORCID

classification cs.SEcs.AI

keywords softwarepackageselectionPythonecosystemmulti-criteriadecision-makingknowledgegraphgenerativeAIdecisionsupportsystemTechnologyAcceptanceModelopen-source

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Package selection in open-source ecosystems has no formal basis: developers choose among many alternatives using popularity signals or whatever a chatbot suggests. This paper tries to change that by treating selection as a Multi-Criteria Decision-Making problem and building PySelect, a system that continuously pulls metadata, usage trends, vulnerabilities, and developer sentiment from GitHub, PyPI, and Stack Overflow into a structured decision model. It then uses a large language model only to translate user needs into queries over that model. The claim, tested on 798,669 Python scripts from 16,887 GitHub repositories and a user study, is that this evidence-based route produces higher-quality, more reproducible recommendations than generic generative-AI tools. The authors argue this gives developers a transparent, auditable way to justify package choices.

What carries the argument

The central object is the decision model: a knowledge-graph-like structure in which packages are linked to domain features and quality attributes. Automated pipelines keep it current by harvesting GitHub repository metadata, PyPI package metadata, usage trends, vulnerability reports, and Stack Overflow developer sentiment. Multi-Criteria Decision-Making supplies the ranking logic, and an LLM acts as the front end that converts user intent into structured queries. The model's work is to turn scattered ecosystem signals into an auditable, reproducible basis for comparing alternatives.

What would settle it

Run PySelect and a generic generative-AI assistant on the same set of real Python package-selection tasks, then follow the recommendations forward: check whether the chosen package installs cleanly, passes the project's test suite, survives a dependency-vulnerability audit, and is still actively maintained after six months. If PySelect does not beat the baseline on these observable outcomes, the claimed quality improvement is not established.

Watch

Extended reading notes

Core claim

On the paper's own account, the central discovery is that the unstructured, popularity-biased process of picking third-party packages can be replaced by a queryable decision model built from continuous, automated data collection. PySelect structures relationships among packages, domain features, and quality attributes as a graph; an LLM interprets a developer's intent in natural language and issues structured queries to that graph. The evaluation compares the system against generative-AI baselines on a corpus of 798,669 Python scripts from 16,887 GitHub repositories and runs a Technology Acceptance Model user study. The reported results are high data-extraction precision, better recommendati

Load-bearing premise

The load-bearing premise is that the evaluation's notion of a 'good' recommendation matches what developers actually need, because the abstract does not say what ground truth was used to decide which packages were the right choices.

Editorial extensions

If this is right

Developers can justify package choices with traceable evidence from live data rather than popularity or habit.
Recommendations become reproducible: the same query over the same data snapshot should return the same package, which generic generative-AI tools cannot guarantee.
Project teams get an early warning channel for vulnerabilities and maintenance risk because the data pipeline explicitly monitors those dimensions.
The evaluation corpus of hundreds of thousands of scripts gives the community a large-scale test bed for future package-selection tools.
Organizations can audit architectural decisions after the fact by inspecting the data and criteria behind each recommendation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test this, but the same architecture should transfer to other package ecosystems such as npm, Maven, or CRAN, because the data-collection logic is ecosystem-agnostic.
An open question the abstract leaves is what 'recommendation quality' was measured against; a stronger follow-up would use downstream outcomes such as whether the chosen package builds, passes tests, and is still maintained months later.
If the LLM-as-interpreter split generalizes, it suggests a broader design principle: use generative models to parse intent rather than to supply facts when the domain already has structured evidence.
A testable extension: freeze a data snapshot and rerun old queries to quantify how much recommendation quality depends on data freshness, isolating the value of the continuous pipeline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a reasonable engineering paper with a real target problem, but the abstract's central claim is uninterpretable as written. The combination of a continuous metadata pipeline, a knowledge graph, MCDM, and LLM intent parsing for Python package selection is a plausible integration that practitioners might find useful. The evaluation corpus is large, which suggests real effort. I credit that.

The soft spots are exactly where the stress-test note lands. "Improved recommendation quality over generative AI baselines" depends entirely on how quality was measured. The abstract never defines what makes a package suitable. If the ground truth comes from packages actually imported in those GitHub scripts, the system is being rewarded for replicating popularity, not for satisfying project constraints. That is not necessarily wrong—replication can be a sensible proxy—but it is circular if the same corpus feeds both the metadata extraction and the evaluation. The baselines are unnamed, so we don't know if they were given the same context. The TAM survey measures perceived usefulness, not actual selection success. None of these are fatal if the full paper handles them, but the abstract gives no reason to trust the claim yet.

I'm not accusing the authors of hiding anything. Abstract-style summaries often elide methodology. Still, the distinction between "extracts metadata with high precision" and "recommends better packages" is exactly the kind of thing that needs to be shown, not asserted.

Who is this for? Researchers and developers working on dependency selection and AI-assisted software engineering. If the evaluation is done cleanly, it is a useful data point. If not, it is another LLM-wrapper demo. A serious referee should ask for the evaluation protocol upfront.

My recommendation: send it to peer review. The flaws I see are fixable with clearer reporting, and the problem is worth serious attention.

Referee Report

4 major / 2 minor

Summary. The paper proposes PySelect, a decision-support framework for Python package selection. It combines automated metadata collection from GitHub, PyPI, and Stack Overflow into a knowledge-graph-based decision model, and uses large language models to interpret user intent and apply Multi-Criteria Decision-Making (MCDM). The evaluation described in the abstract consists of a corpus of 798,669 Python scripts from 16,887 GitHub repositories and a user study based on the Technology Acceptance Model (TAM). The abstract claims high data-extraction precision, improved recommendation quality over generative AI baselines, and positive user evaluations of usefulness and ease of use.

Significance. If substantiated, the work would be a useful contribution to empirical software engineering: it makes package selection explicit and interpretable through an MCDM framework, uses large-scale ecosystem data, and offers a reproducible alternative to opaque generative-AI suggestions. The scale of the corpus and the inclusion of a TAM user study are appropriate strengths. However, the abstract alone does not provide enough protocol detail to verify the central claims; the significance remains conditional on a full methodology description and on a clearly defined recommendation-quality metric.

major comments (4)

[Abstract, sentence beginning 'Results show...'] The central claim 'improved recommendation quality over generative AI baselines' is uninterpretable because neither the recommendation-quality metric nor the ground truth for 'contextually appropriate packages' is defined. If correctness is determined by whether a package is imported in the GitHub scripts that also populate the knowledge graph, the evaluation may largely reward popularity replication rather than suitability. The abstract must state the ground-truth source and the evaluation protocol.
[Abstract, 'generative AI baselines'] The baselines are unnamed and their configuration is not described. A fair comparison requires that the generic generative AI tools receive the same project context, dependency constraints, and interaction opportunities as PySelect. Without this information, the reported improvement cannot be evaluated or reproduced.
[Abstract, 'high data extraction precision' and 'user study'] High data-extraction precision validates the data pipeline, not the quality of package recommendations. The TAM user study measures perceived usefulness and ease of use, which are subjective attitudes, not objective recommendation correctness. The abstract therefore does not provide direct evidence for the headline claim of improved recommendation quality.
[Abstract, evaluation corpus sentence] It is unclear whether the 798,669 Python scripts were used both to construct the decision model and to evaluate the recommendations. If the same corpus serves both roles, the results may be optimistic due to training/evaluation overlap. The abstract should state the data split and the temporal ordering of data collection.

minor comments (2)

[Abstract] The abstract contains no numerical results, effect sizes, or confidence intervals. Quantitative headline results would help readers calibrate 'high' and 'improved'.
[General] As this review is based on the abstract only, section citations are unavailable. A full manuscript should contain an artifact-availability statement and a detailed threats-to-validity section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified from the abstract; evaluation appears externally benchmarked.

full rationale

The abstract describes a data-driven framework that collects package metadata, usage trends, vulnerabilities, and developer sentiment from external sources (GitHub, PyPI, Stack Overflow), structures these into a decision model, and evaluates recommendation quality against 798,669 Python scripts and a Technology Acceptance Model user study, comparing against generative AI baselines. No equations or fitted parameters are presented, and no specific result is shown to reduce definitionally to its inputs. The fact that the recommendation-quality ground truth is not defined in the abstract is a reporting limitation, not evidence of circularity. Without the full methodology, we cannot exhibit a concrete reduction where an output is equivalent to an input by construction. The evaluation relies on external baselines and user perceptions, which are independent of the framework's internal model construction. Therefore no circularity step is established; the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

This is an abstract-only review, so the ledger lists only what can be inferred. No equations or fitted constants appear in the abstract. The free parameters are the weighting scheme and data thresholds of the framework, neither of which is disclosed in the abstract. The axioms are domain-level premises about LLM reliability, data quality, and the validity of the TAM-based user study.

free parameters (2)

MCDM criteria weights
The abstract describes a decision model that weighs quality attributes (e.g., popularity, security, maintainability) but does not state how these weights are set. If chosen by researchers or users, they are free parameters affecting recommendation outcomes.
Data collection thresholds and filters
The automated pipelines that collect metadata from GitHub, PyPI, and Stack Overflow must use inclusion/exclusion thresholds (e.g., repository size, star counts, update recency) that shape the corpus of 798,669 scripts; these are not specified in the abstract.

assumptions (3)

domain assumption LLMs can faithfully map user intent to structured queries against the decision model.
PySelect relies on LLMs to interpret user intent and query the model; if this mapping is unreliable, recommendation quality degrades. The abstract does not validate this mapping separately.
domain assumption The metadata collected from GitHub, PyPI, and Stack Overflow is accurate and sufficiently comprehensive to assess package suitability.
All recommendation inputs are derived from these data sources; the abstract reports no independent validation of data quality or coverage.
domain assumption The Technology Acceptance Model survey responses are a valid proxy for actual usefulness and adoption of the tool.
TAM measures self-reported usefulness and ease of use, not objective selection success; using it as the user-study outcome assumes this connection.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach." pith.science (2026). https://pith.science/paper/7JQEEUIP

@misc{pith2026250805693,
  author       = {Pith},
  title        = {Pith review of: Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JQEEUIP}},
  note         = {Machine review of arXiv:2508.05693}
}

read the original abstract

Selecting third-party software packages in open-source ecosystems like Python is challenging due to the large number of alternatives and limited transparent evidence for comparison. Generative AI tools are increasingly used in development workflows, but their suggestions often overlook dependency evaluation, emphasize popularity over suitability, and lack reproducibility. This creates risks for projects that require transparency, long-term reliability, maintainability, and informed architectural decisions. This study formulates software package selection as a Multi-Criteria Decision-Making (MCDM) problem and proposes a data-driven framework for technology evaluation. Automated data pipelines continuously collect and integrate software metadata, usage trends, vulnerability information, and developer sentiment from GitHub, PyPI, and Stack Overflow. These data are structured into a decision model representing relationships among packages, domain features, and quality attributes. The framework is implemented in PySelect, a decision support system that uses large language models to interpret user intent and query the model to identify contextually appropriate packages. The approach is evaluated using 798,669 Python scripts from 16,887 GitHub repositories and a user study based on the Technology Acceptance Model. Results show high data extraction precision, improved recommendation quality over generative AI baselines, and positive user evaluations of usefulness and ease of use. This work introduces a scalable, interpretable, and reproducible framework that supports evidence-based software selection using MCDM principles, empirical data, and AI-assisted intent modeling.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection
cs.SE 2026-08 conditional novelty 6.0 of 10

An explainable multi-criteria system ranks 71,274 Hugging Face models using metadata, functional features, and community-perceived quality, matching commercial LLM recommenders on coverage while exposing criterion-lev...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.