Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends

Armstrong Foundjem; Florian Tambon; Foutse Khomh; Gianolli Dorcelus; Mina Taraghi

RECORD

Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends

Mina Taraghi, Gianolli Dorcelus, Armstrong Foundjem, Florian Tambon, Foutse Khomh

Software Engineering · Submitted 24 Jan 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2401.13177 (version 1)

Pith IDHGM75V6U, Pith's permanent record of this paper and its review

CitationArmstrong Foundjem, Florian Tambon, Foutse Khomh, Gianolli Dorcelus, Mina Taraghi. "Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends." arXiv:2401.13177 (2024). https://pith.science/paper/HGM75V6U

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,863 input tokens.

What the abstract itself states, as probabilities: claims first 0.21 · states limitation 0.17 · human participants 0.35 · controlled comparison 0.03 · randomized assignment 0.01 · states quantitative result 0.07 · compares with prior results 0.09 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs. Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking. To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub. Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community. We then conduct a quantitative study to track model-type trends and model documentation evolution over time. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding. We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them. Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users. Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs.	background 1.00	0.04	0.02	asserted 1.00
s1	Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking.	gap 1.00	0.09	0.02	asserted 1.00
s2	To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub.	method 0.70	0.96	0.02	asserted 1.00
s3	Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community.	contribution 0.81	0.96	0.09	asserted 1.00
s4	We then conduct a quantitative study to track model-type trends and model documentation evolution over time.	method 0.94	0.96	0.02	asserted 0.94
s5	Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding.	result 1.00	0.94	0.40	asserted 1.00
s6	We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them.	result 1.00	0.95	0.07	asserted 0.97
s7	Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users.	result 0.99	0.94	0.12	asserted 1.00
s8	Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.	contribution 0.66	0.88	0.63	asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from analysis of existing data 0.76.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs. Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking. To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub. Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community. We then conduct a quantitative study to track model-type trends and model documentation evolution over time. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding. We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them. Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users. Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

6 arXiv papers cite this in 11 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 6 sentences without a confident label

Contests it 1 paper

Therefore, the significant relationship between model card presence and observed for , and models contradicts the overall lack of model card documentations reported in [ref] and [ref] .

Towards Semantic Versioning of Open Pre-trained Language Model Releases on Hugging Face · Result · contest 0.72

Compares with it 1 paper

Our analysis suggests a potential improvement in the prevalence of model cards specifically for on compared to [ref] and [ref] broader findings.

Towards Semantic Versioning of Open Pre-trained Language Model Releases on Hugging Face · Result · 0.99

Cites it as background 3 papers

Previous studies have investigated challenges in PTM adoption from a broad ecosystem perspective [ref] .

When AI Models Become Dependencies: Studying the Evolution of Pre-Trained Model Reuse in Downstream Software Systems · Introduction · 0.95
These investigations cover a wide range of perspectives, including machine learning [ref] , software engineering [ref] , and social computing [ref] .

Exploring the Use of Abusive Generative AI Models on Civitai · Related Work · 0.96
The mainstream artificial intelligence community has experienced a surge in the availability of large-scale open-source classifiers [ref] , catering to diverse end-users with varying use cases.

Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation · Introduction · 0.92

What it cites · 13 sentences without a confident label

Cites as support 2 papers

Yet, training DNNs from scratch is usually very time-consuming, computationally expensive, and has a significant carbon footprint [ref] [ref] [ref] .

Quantifying the Carbon Emissions of Machine Learning · Introduction · 0.97
Yet, training DNNs from scratch is usually very time-consuming, computationally expensive, and has a significant carbon footprint [ref] [ref] [ref] .

Carbon Emissions and Large Neural Network Training · Introduction · 0.97

Cites as background 2 papers

Pan et al. [ref] studied the GitHub repositories of popular models supported by HF transformers, analyzing reported bugs to create a detailed taxonomy, and uncovering root causes and implications for model reuse.

An Empirical Study on the Bugs Found while Reusing Pre-trained Natural Language Processing Models · PTM Reuse and Model Hubs · 0.88
In another recent work, Jiang et al. [ref] explored the naming conventions and defects in PTM packages in different model hubs including HF, and proposed a method to detect the defects.

"I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency · PTM Reuse and Model Hubs · 0.94

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.