REVIEW 4 major objections 4 minor 27 references
HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that foundation-model selection can be treated as an auditable multi-criteria decision problem, and that a deterministic weighted-sum recommender over an automatically built knowledge base matches commercial LLM-based…
desk verdict An honest, reproducible systems paper whose 'comparable to commercial LLMs' claim rests on small-sample point estimates, but whose integrated pipeline is a real and useful contribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is a Weighted Sum Model (WSM/SAW): each candidate model $m_i$ receives a score $R(m_i,q)=\sum_j w_j \cdot s_{ij}$, where $w_j$ are user-derived criterion weights that sum to 1 and $s_{ij}\in[0,1]$ are normalized criterion scores. Perceived quality scores are aggregated from community review snippets as $S_{\text{fuzzy}}=H/(L+H)$, the ratio of positive over directional reviews, and a quality attribute is scored only when at least three evidence-bearing reviews exist. A graph-structured knowledge base stores models, features, quality attributes, and source snippets, but the ranking logic itself is the additive model, whose per-term products $w_j\cdot s_{ij}$ are what make the recommendations inspectable and tunable.
What would settle it
Build a ground-truth set from benchmark leaderboards and deployment logs instead of literature-reported choices, run the same or an expanded set of scenarios, and check whether model-level Coverage@10 stays near 0.61; if it falls toward the metadata-only ablation level of 0.09 or the no-functional-features level of 0.23, the literature ground truth was carrying the result.
Extended reading notes
Core claim
The central claim is that repository-scale foundation-model selection can be operationalized as an explicit multi-criteria decision problem without sacrificing recommendation quality. HugSelect builds a unified knowledge base from three evidence families: structured repository metadata, functional capabilities extracted from model-card text, and ISO/IEC 25010-inspired perceived quality attributes derived from community reviews. It ranks candidates with a Weighted Sum Model (WSM/SAW), computing $R(m_i,q)=\sum_j w_j \cdot s_{ij}$ and decomposing each score into criterion-level contributions for inspection. The evaluation shows model-level Coverage@10 of 0.61 and family-level Coverage@10 of 0.91, with no significant overall difference in ranking quality against four commercial zero-shot LLM recommenders, and an ablation shows that removing functional features drops model-level coverage from 0.61 to 0.23 while removing both functional and quality features drops it to 0.09. The value is not that HugSelect beats chatbots, but that competitive retrieval comes with score decomposition, feature-level traceability, and comparative trade-off views that opaque conversational systems do not provide.
Load-bearing premise
The load-bearing premise is that the model or model family chosen in a peer-reviewed paper, verified through its linked GitHub repository, is a valid proxy for what a good recommendation should retrieve; if those literature-derived labels are biased toward popular or convenient models, the Coverage@10 and NDCG@10 comparisons against commercial LLMs measure agreement with proxy labels rather than true selection quality.
Editorial extensions
If this is right
- A deterministic, auditable recommender can substitute for opaque conversational advice in foundation-model selection without a measurable drop in top-10 retrieval quality.
- Functional capability extraction from model cards is the main driver of exact-model retrieval, so teams building such systems should invest in model-card and README parsing before community sentiment.
- Family-level coverage near 0.91 means the system is particularly useful for surfacing substitute variants such as base, quantized, and fine-tuned checkpoints rather than a single canonical model.
- Removing both functional and quality features drops model-level coverage to 0.09, showing that metadata-only or popularity-based ranking is far weaker than multi-source evidence.
- Because weights are explicit and adjustable, practitioners can audit and adapt rankings to project constraints such as license, format, and reliability without re-prompting a black-box system.
Reading between the lines
- An implication the authors leave implicit is that the same WSM/SAW shell could port to other model hubs or private registries, since the framework is repository-agnostic in principle; the open question is whether the extraction pipelines transfer without retraining.
- The measured parity with LLM chatbots is time-bound: the baselines were zero-shot with no prompt optimization, so a conversational recommender augmented with retrieval or fine-tuning could plausibly close or reverse the gap.
- A natural testable extension is to replace the fuzzy sentiment ratio with benchmark-based quality evidence and see whether family-level discrimination improves; the paper lists richer evidence as future work, and the modular design makes this straightforward.
- The jump from 0.61 model-level coverage to 0.91 family-level coverage reflects the paper's choice to treat family members as substitutes; a user who needs an exact checkpoint would see lower effective performance, which suggests a UI that separates exact matches from family matches.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HugSelect, an explainable multi-criteria decision-support framework for selecting foundation models from Hugging Face. It builds a knowledge base of 71,274 models by combining repository metadata, functional features extracted from model cards, and quality attributes derived from community feedback via LLM-based mapping. Rankings are produced with a weighted sum model with criterion-level score decomposition. The evaluation includes extraction pipeline validation, comparison with four commercial LLM-based recommenders on 44 literature-derived scenarios, ablation, and a 10-participant user study. The central claim is that HugSelect achieves recommendation quality comparable to commercial LLM recommenders while providing transparent reasoning.
Significance. If the results hold, the paper demonstrates that a deterministic, explainable system with an automatically constructed knowledge base can be competitive with LLM chat recommenders on a realistic selection-retrieval task. The strengths include a publicly available replication package, a repository-scale knowledge base, an external literature-based evaluation benchmark, and an honest and explicit treatment of threats to validity. The functional-feature pipeline validation uses a human-in-the-loop protocol that partially avoids circularity, and the ablation study provides clear evidence of the contribution of functional features. However, the quality-attribute mapping validation is partially circular, and the comparative evaluation at n=44 lacks confidence intervals, so the strength of the central claim is limited.
major comments (4)
- [Section 5.2.2, Table 7] The reported 0.84 accuracy for quality-attribute mapping is validated against a reference set formed by three LLMs (labels appearing in at least two outputs, refined by a human annotator), while the mapping pipeline itself uses LLM prompting. This is a circular evaluation for the mapping step: the pipeline is measured against ground truth generated by the same kind of model it uses. The abstract and Section 5.2.2 should either report a human-only validation or explicitly qualify the 0.84 figure as an agreement rate among LLMs, not as accuracy against independent human labels.
- [Section 5.3, Table 11] The comparative claim of 'no significant overall differences' rests on 44 cases with point estimates and no confidence intervals. The observed differences (model Coverage@10 0.61 vs. Claude's 0.64; family-level 0.91 vs. 0.93) are well within sampling noise, and the only significant McNemar comparison is against Gemini. The manuscript should report confidence intervals for Coverage@10 and NDCG@10, and rephrase 'comparable' as 'no statistically significant difference was detected', with an explicit discussion of the low statistical power at n=44. Without this, the reader cannot gauge the strength of the evidence for equivalence.
- [Section 5.3.1, Ground Truth Construction] The 44 literature-derived proxy labels are a load-bearing condition for the entire comparative evaluation. The paper acknowledges that 'other suitable alternatives may exist' (Section 5.6), but it does not examine whether the ground-truth models are systematically biased toward popular or well-documented models, which would inflate Coverage@10 for any system that reads model-card text. I recommend adding an analysis of the popularity and documentation-quality distribution of the ground-truth models, and a sensitivity check of Coverage@10 when the top-k most popular models are removed from the evaluation.
- [Section 3.6 and Section 5.5] The framework's sensitivity to the MoSCoW criterion weights is a core design feature but is never evaluated. Section 3.6 mentions a sensitivity analysis under ±20% weight perturbations, yet no results of such an analysis appear in Section 5. The abstract's claim of 'stable' reasoning is thus unsupported. Either report the sensitivity results or explicitly defer them to future work in the evaluation section.
minor comments (4)
- [Table 11] The HugSelect row appears malformed ('0.61-0.91-0.37 0.74') and does not align with the six-column header; please format the row to match the header, using an em dash or 'N/A' for Overlap@10.
- [Section 2.2] Duplicate word in the text: 'perceived perceived quality attributes'.
- [Section 5.2.1] The two paragraphs describing functional-feature ground-truth construction are inconsistent: the first says features appearing in at least two LLM outputs were retained and validated by a human expert, while the second says a feature was included only when identified by the human annotator and confirmed by at least one LLM. Please clarify which protocol was actually used.
- [Throughout] Several sentences start with lowercase 'foundation-models' (e.g., the first sentence of Section 1 and Section 2.2); please fix capitalization at sentence starts.
Circularity Check
Central comparative claim is externally grounded; the 0.84 quality-mapping accuracy is partly self-confirmed because both the pipeline and its reference set are LLM-based, though human refinement and the external 44-case benchmark keep the paper from being circular overall.
-
self definitional
[Section 4.2.1 and Section 5.2.2, Table 7]
"The filtered review snippets are then mapped to ISO/IEC 25010-inspired quality characteristics... Mapping is performed through LLM prompting that interprets semantic cues in the review text and assigns the most relevant quality dimension. ... Three LLMs independently labeled each review; labels appearing in at least two outputs formed the reference set, which was then refined by a human annotator."
The quality-attribute mapping pipeline being validated is an LLM-prompting procedure, and the reference set against which its 0.84 accuracy is computed is produced by the same kind of procedure: three LLMs labeling the same reviews, with 2-of-3 agreement as the reference, then human refinement. Thus the reported accuracy largely measures agreement between an LLM-based mapper and an LLM-based consensus, not agreement with an independent ground truth. The human-in-the-loop step prevents this from being fully self-confirming, but the statement that the pipeline 'achieved accuracy of 0.84 for quality attribute mapping' is partly constructed by the very method under evaluation.
full rationale
The paper's main claim—that HugSelect achieves recommendation quality comparable to commercial LLM recommenders—is tested against an external benchmark: 44 literature-derived selection scenarios with model/family-level coverage and NDCG metrics. That evaluation is not circular: the proxy ground truth comes from peer-reviewed papers and GitHub repositories, not from HugSelect's own output, and the authors explicitly acknowledge its proxy nature in the 'Evaluation Caveats' section and Section 5.6. The ablation study and statistical tests are self-contained empirical comparisons, not fitted predictions. The self-citations to Farshidi and Joonbakhsh are methodological lineage (e.g., 'adapted from the multi-criteria decision-making framework proposed by Farshidi (2020)') and are not load-bearing for the quantitative results. The only material circularity is the RQ1 validation of the quality-attribute mapping: the pipeline uses LLM prompting, and its reference labels are derived from 2-of-3 LLM agreement (further refined by a human). This makes the headline 0.84 mapping accuracy partly self-confirmatory, and the paper itself acknowledges similar concerns for extraction validation in Section 5.2.1. Because the central competitive claim rests on an external benchmark and the circular step is confined to a supporting pipeline-validation metric, the overall circularity score is 3 rather than higher.
Assumptions & free parameters
free parameters (3)
- Minimum community evidence threshold =
L + H >= 3
- Evaluation cutoff k =
10
- MoSCoW criterion weights =
not fixed; assigned per query by LLM
assumptions (5)
- domain assumption Literature-reported model choices are valid proxy ground truth for recommendation quality.
- domain assumption Community review snippets can be reliably coded into ISO/IEC 25010 quality attributes by LLM prompting.
- domain assumption Perceived quality from community feedback approximates quality relevant for selection.
- domain assumption WSM/SAW additive preference structure is appropriate for model selection.
- domain assumption The repository snapshot and community platforms provide representative evidence at scale.
Cite this review
Pith. "Pith review of HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection." pith.science (2026). https://pith.science/paper/ZEZEGVXW
@misc{pith2026260808069,
author = {Pith},
title = {Pith review of: HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZEZEGVXW}},
note = {Machine review of arXiv:2608.08069}
}
read the original abstract
Foundation models are increasingly reused as software components, making model selection a critical software-engineering decision. Current model hubs primarily support discovery through popularity metrics, often neglecting functional capabilities, operational constraints, and community-perceived quality. We argue that foundation-model selection should be treated as an explicit, auditable software-component selection task rather than as keyword search, popularity ranking, or opaque conversational advice. This paper proposes HugSelect, an explainable decision-support framework for foundation-model selection. HugSelect builds a knowledge base of 71,274 models by combining repository metadata, extracted functional capabilities, and perceived quality attributes derived from community discussions into a unified pipeline. It ranks candidate models using a weighted additive model that exposes criterion-level score decompositions. We evaluated HugSelect through pipeline validation, comparative case studies against four commercial LLM-based recommendation systems (44 scenarios), fine-grained ablation, and an exploratory user study (n = 10). Extraction pipelines achieved an F1 score of 0.801 for functional features and an accuracy of 0.84 for quality-attribute mapping. HugSelect achieved a model-level Coverage@10 of 0.61 and family-level Coverage@10 of 0.91, showing recommendation quality comparable to that of the evaluated commercial systems, with no significant overall differences in ranking quality, while providing stable, traceable, and inspectable reasoning. Ablation confirmed that functional features were the main driver of retrieval accuracy, and preliminary user feedback suggests that the framework is useful and intuitive.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[6]
Gen-AI: Artificial intelligence and the future of work. International Monetary Fund. Chakrabortty,R.K.,Abdel-Basset,M.,Ali,A.M.,2023. Amulti-criteriadecisionanalysismodelforselectinganoptimumcustomerservicechatbot under uncertainty. Decision Analytics Journal 6, 100168. Chen, Q., Huang, K., Zhou, X., Luo, W., Cui, Y., Cheng, G.,
work page 2023
-
[7]
URL:https://huggingface.co/deepseek-ai/DeepSeek-R1
Deepseek-r1: Reasoning-optimized language model. URL:https://huggingface.co/deepseek-ai/DeepSeek-R1. accessed: 2025-01-15. Di Sipio, C., Rubei, R., Di Rocco, J., Di Ruscio, D., Nguyen, P.T.,
work page 2025
-
[8]
Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
Learning to rank pre-trained vision-language models for downstream tasks. arXiv preprint arXiv:2412.20682 . Drissi, N., El-Kassabi, H., Serhani, M.,
-
[9]
Decision Analytics Journal doi:10.1016/j.dajour.2024.100523
A multi-criteria decision analysis framework for evaluating deep learning models in healthcare research. Decision Analytics Journal doi:10.1016/j.dajour.2024.100523. Farshidi, S.,
arXiv 2024
-
[11]
Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach
Empirical evaluation of ai-assisted software package selection: A knowledge graph approach. arXiv preprint arXiv:2508.05693 . Faulconbridge, J., Sarwar, A., Spring, M.,
-
[12]
arXiv preprint arXiv:2506.03013
How do pre-trained models support software engineering? an empirical study in hugging face. arXiv preprint arXiv:2506.03013 . GoogleDeepMind,2023.Gemini:Afamilyofhighlycapablemultimodalmodels.URL:https://deepmind.google/technologies/gemini/. technical Report. Hevner, A.R., March, S.T., Park, J., Ram, S.,
arXiv 2023
-
[14]
URL:https://huggingface.co/docs/hub/api
Hugging face hub api. URL:https://huggingface.co/docs/hub/api. accessed: 2024-01-15. Joonbakhsh et al.:Preprint submitted to ElsevierPage 28 of 29 HugSelect: Explainable foundation-model selection Jain, S.M.,
work page 2024
-
[15]
URL: https://ceur-ws.org/Vol-2912/paper3.pdf
SearchSECO: A worldwide index of the open source software ecosystem, in: Proceedings of the 19th Belgium-Netherlands Software Evolution Workshop (BENEVOL 2020), CEUR-WS.org. URL: https://ceur-ws.org/Vol-2912/paper3.pdf. Joonbakhsh, A., Rostami, A., Kamalinia, A., Nazeri, A., Khunjush, F., Tekinerdogan, B., Farshidi, S.,
work page 2020
Show all 27 references
-
[16]
arXiv preprint arXiv:2512.11984
Evidence-driven decision support for ai model selection in research software engineering. arXiv preprint arXiv:2512.11984 . Leščinskait˙e, P., Paulavičius, R., Filatovas, E.,
-
[17]
Li, N., Farshidi, S., Bianchi, R., Koulouzis, S., Zhao, Z.,
URL:https://www.mdpi.com/2227-7390/14/9/1536, doi:10.3390/math14091536. Li, N., Farshidi, S., Bianchi, R., Koulouzis, S., Zhao, Z.,
-
[20]
arXiv preprint arXiv:2402.06196
Large language models: A survey. arXiv preprint arXiv:2402.06196 . Olabanjo,O.,Honenberger,P.,2026. Mcda4ai:Aframeworkformanagingn>2criteriaproblemsindecisionsaboutartificialintelligence. Array29, 100723.URL:https://www.sciencedirect.com/science/article/pii/S2590005626000469,d...
2026 arXiv
-
[21]
URL:https://www.reddit.com/dev/api/
Reddit api documentation. URL:https://www.reddit.com/dev/api/. accessed: 2024-01-15. Ricci,F.,Rokach,L.,Shapira,B.,2010. Introductiontorecommendersystemshandbook,in:Recommendersystemshandbook.Springer,pp.1–35. Seker, S., Kahraman, C.,
2024
-
[22]
SoftComputing-AFusionofFoundations,MethodologiesandApplicationsURL:https://www.researchsquare.com/ article/rs-709398/latest.pdf, doi:10.1007/s00500-021-06469-8
A Pythagorean cubic fuzzy methodology based on TOPSIS and TODIM methods and its application to software selectionproblem. SoftComputing-AFusionofFoundations,MethodologiesandApplicationsURL:https://www.researchsquare.com/ article/rs-709398/latest.pdf, doi:10.1007/s00500-021-064...
-
[23]
URL:https://api.stackexchange.com/docs
Stack exchange api v2.3. URL:https://api.stackexchange.com/docs. accessed: 2024-01-15. Suryani, M.A., Karmakar, S., Mathiak, B.,
2024
-
[24]
arXiv preprint arXiv:2302.13971
Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 . Triantaphyllou, E.,
-
[27]
arXiv preprint arXiv:2408.02920
A taxonomy of architecture options for foundation model-based agents: Analysis and decision model. arXiv preprint arXiv:2408.02920 . Joonbakhsh et al.:Preprint submitted to ElsevierPage 29 of 29
-
[2004]
MIS quarterly , 75–105
Design science in information systems research. MIS quarterly , 75–105. Honnibal,M.,Montani,I.,2017. spacy2:Naturallanguageunderstandingwithbloomembeddings,convolutionalneuralnetworksandincremental parsing, To appear. Hugging Face,
2017
-
[2006]
European Journal of operational research 169, 1–29
Analytic hierarchy process: An overview of applications. European Journal of operational research 169, 1–29. Venkatesh,V.,Davis,F.D.,2000. Atheoreticalextensionofthetechnologyacceptancemodel:Fourlongitudinalfieldstudies. Managementscience 46, 186–204. Wohlin, C.,
2000
-
[2012]
Physics reports 519, 1–49
Recommender systems. Physics reports 519, 1–49. Martin,W.,Sarro,F.,Jia,Y.,Zhang,Y.,Harman,M.,2017. Asurveyofappstoreanalysisforsoftwareengineering. IEEETransactionsonSoftware Engineering 43, 817–847. doi:10.1109/TSE.2016.2630689. Mienye, I.D., Jere, N., Obaido, G., Ogunruku, O...
2017
-
[2014]
Guidelines for snowballing in systematic literature studies and a replication in software engineering, in: Proceedings of the 18th international conference on evaluation and assessment in software engineering, pp. 1–10. Zhang,Y.,Ding,H.,Shui,Z.,Ma,Y.,Zou,J.,Deoras,A.,Wang,H.,2...
-
[2020]
Multi-criteria decision-making in software production. Ph.D. thesis. Utrecht University. Farshidi,S.,Jansen,S.,Deldar,M.,2021.Adecisionmodelforprogramminglanguageecosystemselection:Sevenindustrycasestudies.Information and software technology 139, 106640. Farshidi, S., Jansen, ...
2021
-
[2021]
arXiv preprint arXiv:2108.07258
On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 . Cao, X., Shi, Y., Yu, H., Wang, J., Wang, X., Yan, Z., Chen, Z.,
-
[2022]
Context-aware notebook search in a jupyter-based virtual research environment, in: 2022 IEEE 18th International Conference on e-Science (e-Science), IEEE. pp. 393–394. doi:10.1109/eScience55777.2022.00054. Liu, M., Zhao, C., Peng, X., Yu, S., Wang, H., Sha, C.,
2022
-
[2023]
arXiv preprint arXiv:2310.18689
Foundational models in medical imaging: A comprehensive survey and future vision. arXiv preprint arXiv:2310.18689 . Behzadian, M., Otaghsara, S.K., Yazdani, M., Ignatius, J.,
-
[2024]
Procedia Computer Science 233, 909–918
Machine learning as a service cloud selection: An mcdm approach for optimal decision making. Procedia Computer Science 233, 909–918. URL:https://www.sciencedirect.com/science/article/pii/S1877050924006409, doi:https://doi.org/10.1016/j.procs.2024.03.280. 5th International Conf...
-
[2025]
ACMTransactionsonSoftwareEngineeringandMethodology34,23:1–23:35
Decision support model for selecting the optimal blockchain oracle platform:Anevaluationofkeyfactors. ACMTransactionsonSoftwareEngineeringandMethodology34,23:1–23:35. doi:10.1145/3697011. Alsalem,M.A.,Alamoodi,A.H.,Albahri,O.S.,Albahri,A.S.,Martínez,L.,Yera,R.,Duhaim,A.M.,Shar...
-
[2026]
URL:https://doi.org/10.17632/9kxtkvyv5m.2, doi:10.17632/9kxtkvyv5m.2
Hugselect dataset and replication package. URL:https://doi.org/10.17632/9kxtkvyv5m.2, doi:10.17632/9kxtkvyv5m.2. Adomavicius, G., Manouselis, N., Kwon, Y.,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.