Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Model release and model reuse are misaligned: a 145-person survey shows producers and consumers systematically disagree on where metadata belongs, why lineage matters, and which governance tools help.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:51 UTC pith:JBP6S6VP

load-bearing objection A genuinely first dual-perspective survey with a useful 16/20 metadata-placement finding, but the statistical scaffolding for 'systematic' is thinner than the prose, and the ethics statements contradict each other. the 3 major comments →

arxiv 2607.21738 v1 pith:JBP6S6VP submitted 2026-07-23 cs.SE

When Model Release Meets Model Reuse: Producer-Consumer Misalignment in Hugging Face

Adekunle Ajibode , Oussama Ben Sghaier , Bram Adams , Ahmed E. Hassan This is my paper
classification cs.SE
keywords AI supply chainmodel reuseHugging Facepre-trained language modelsdocumentation misalignmentmodel lineagemodel governancepractitioner survey
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that persistent problems in the pre-trained language model (PTLM) supply chain—inconsistent release practices, incomplete metadata, and weak lineage tracing—are symptoms of a systematic misalignment between model producers and the developers who reuse those models. It is the first dual-perspective survey of 50 Hugging Face producers and 95 GitHub consumers, asking both cohorts the same core questions across discovery, documentation, lineage, and governance. The central claim is that producers and consumers rely on the same documentation artifacts but disagree on where critical metadata should live: for 16 of 20 metadata types, the two groups' placement choices differed significantly. Lineage tracing is rare (27.9% of producers, 31.4% of consumers) and serves different goals—producers trace for provenance and compliance, consumers for quality and reliability. If this is right, fixing the AI supply chain means realigning roles and incentives, not just adding more documentation.

Core claim

The discovery is that producer-consumer misalignment is systematic, not incidental. Across 20 metadata types, producers and consumers were asked where each type of information should be stored (repository name, README, model card, configuration file, tags, or elsewhere) or found; for 16 of the 20, the two cohorts' responses differed with statistical significance and medium-to-large effect sizes. Example: producers put model names in the repository name (56.1%) while consumers expect them in model cards (40.6%); producers put training configuration in READMEs (67.5%) while consumers expect configuration files (42.0%). The paper argues this 'misplaced rather than missing' phenomenon explains w

What carries the argument

The pair of matched survey instruments with identical core questions is the measurement device; the explanatory mechanism is the role split between release and reuse. The load-bearing construct is metadata-location mismatch, quantified by per-metadata-type Chi-square tests (with Benjamini-Hochberg correction and Cramer's V effect sizes) across 20 metadata types, and by qualitative coding of 614 open-ended responses. This turns 'missing documentation' into a more precise claim: information is present but placed where the other role does not expect it.

Load-bearing premise

The headline rates are sample statistics from 145 self-selected respondents; if practitioners with painful reuse failures or strong opinions were over-represented, the measured misalignment would be overstated.

What would settle it

A telemetry study that records where consumers actually look for each metadata type on Hugging Face—and whether they find it—would test the claim: if search logs show consumers reliably find producer-placed metadata, or if a larger probability sample fails to reproduce significant differences in 16 of 20 metadata placements, the systematic-misalignment thesis would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Even when documentation exists, platforms should validate where metadata is placed, not just whether it is present; model-card templates alone will not resolve the mismatch.
  • Producer-preferred governance (naming conventions, semantic versioning, lineage graphs) and consumer-preferred governance (model-card templates, dependency disclosure) are complementary and should be bundled rather than treated as alternatives.
  • Lineage tracing will remain below a third of practitioners unless base-model documentation and cross-platform provenance aggregation improve; consumers will continue to substitute reputation, external cross-referencing, and empirical testing for direct verification.
  • AIBOM and SPDX-profile tools will face adoption resistance as long as the underlying metadata they depend on is incomplete or unstructured; the collection burden, not the generation tooling, is the stated bottleneck.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: agreement on where metadata belongs could be turned into a platform-level health metric; publishing alignment scores by metadata type would create pressure to converge and a concrete benchmark for improvement.
  • Editorial inference: the finding that consumers abandon models rather than chase missing documentation suggests documentation quality is a classic externality—producers bear the cost of writing it, consumers bear the cost of its absence—so default-enabled templates or automated metadata capture may correct the under-investment more effectively than guidelines.
  • Editorial inference: a testable extension would track real consumer search behavior on Hugging Face; if telemetry shows consumers finding producer-placed metadata without friction, the disagreement is about stated preference rather than operational harm, which would change which fixes matter.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports the first dual-perspective survey of Hugging Face PTLM producers (n=50) and GitHub-based PTLM consumers (n=95), covering model discovery, documentation practices, lineage tracing, and governance mechanisms. It finds that the two groups rely on the same documentation artifacts but systematically disagree on where metadata should be stored, that lineage tracing beyond the immediate parent is rare in both groups (27.9% vs. 31.4%) and pursued for different reasons, and that governance preferences differ by role. The authors interpret these results as evidence that producer–consumer misalignment is a systematic, human-driven contributor to known PTLM supply-chain problems.

Significance. If correct, the paper supplies a missing human-ecosystem layer to previous repository-mining studies of Hugging Face metadata and model supply chains. The design has clear strengths: two parallel instruments with shared core questions, piloting with active maintainers, inter-rater reliability reporting (Krippendorff's alpha 0.678–1.000), a replication package, and use of Cramér's V for effect sizes. The main inferential bridge, however, is the §4.2 claim that 16 of 20 metadata-location comparisons are statistically significant; this is currently not supported by reported per-item statistics, and its validity determines whether the word 'systematic' is justified. The response-rate and representativeness limitations also need to be confronted more directly before the population-level framing can be accepted.

major comments (3)
  1. [§4.2, Figure 4.2, §3.4] The load-bearing sentence 'for 16 out of the 20 types of model information ... the survey responses were statistically significantly different' is not backed by the per-item test statistics, p-values, effect sizes, or contingency tables. Since these are single-choice items and §3.4 says raw p-values were used for single-choice questions, no multiple-comparison correction was applied across the 20 tests; at α=0.05 one would expect roughly one spurious significant result by chance alone. More importantly, with seven location categories and cohort sizes around 41–50 and 70, many cells in Figure 4.2 are at 0–10%, so expected counts below 5 are plausible and Pearson chi-square may be unreliable. Please report exact or Monte-Carlo tests, per-item p-values, and Cramér's V, and apply a correction across all 20 tests. If the number of significant comparisons drops materially, the 'systematic rath
  2. [§3.3, §6, Figure 3.1] The paper's generalizing claims rest on a 5.72% overall response rate from cold-email recruitment, with a sampling frame restricted to contributors who had public email addresses and, in Phase 2, recent activity. Non-response bias could plausibly inflate the measured misalignment if aggrieved practitioners were more likely to respond. The mitigations offered in §6 — qualitative saturation, diversity of roles, and 37.9% of consumers being neutral on selection ease — do not quantify the direction or magnitude of non-response bias. I recommend adding a sensitivity analysis (e.g., early- vs. late-responder comparisons on key outcomes) or a formal comparison of respondent demographics with the full sampling frame, and, at minimum, explicitly framing the conclusions as pertaining to reachable active practitioners rather than to the full Hugging Face/GitHub populations.
  3. [§4.2, RQ2 text] The sentence 'The 40% who find selection easy despite low documentation completeness are those who have developed effective workarounds, not those who actually find the documentation adequate' is not a direct survey result. No cross-tabulation is reported between consumers' selection-ease ratings and their reported workaround use, so the claim is an unsupported causal inference. Either add the relevant cross-tab or remove this interpretation from the results section.
minor comments (5)
  1. [Back matter (Ethical Approval, Informed Consent)] The manuscript states in §3.1 that the study 'received ethics approval from our university prior to data collection,' yet the back matter says 'This study does not involve human participants or animals' and 'No human subjects were involved in this study.' This is a direct contradiction and must be corrected.
  2. [§6, Figure 3.1] Response rates are inconsistent: §6 says '7.17% for Hugging Face and 5.07% for GitHub,' whereas Figure 3.1 reports phase-specific rates and an overall 5.72%; 95/1,838 = 5.17%, not 5.07%. Please reconcile the numbers.
  3. [Figure 4.2] The producer and consumer panels have different row sets: the consumer panel includes 'Concluded license' while the producer panel does not. If the surveys differed, this should be stated; if the figures are misaligned, the 20-type count needs to be checked.
  4. [Figure 4.1] The figure is very dense: multiple codebook categories, percentages, and bracketed governance comparisons are packed into one image. Splitting it by RQ and increasing font sizes would make the results usable.
  5. [Author Contributions] The contributions list includes 'Keheliya Gallaba,' who is not listed among the manuscript authors on the title page; please align the contributor list.

Circularity Check

0 steps flagged

No circularity: the survey findings are self-contained empirical observations; self-citations are background or corroborative, not load-bearing.

full rationale

The paper's support chain is: survey responses -> descriptive comparisons -> chi-square tests -> interpretation of systematic misalignment. No step is derived from an equation, fitted parameter, or normalized ansatz. The load-bearing '16 out of 20' claim in §4.2 is a summary of independent contingency-table tests on the survey responses themselves, not a quantity assumed in advance; the producers' and consumers' answers are separate empirical inputs and the comparison is the result. No 'prediction' is made from fitted values, and no mathematical result is imported from prior work. The self-citations (e.g., Ajibode et al. 2025, 2026; Jewitt et al. 2025) appear as background literature or as post-hoc triangulation in §6 ('results align with repository mining studies'); removing them would not change the survey estimates or the misalignment findings, so they are not load-bearing. There is no uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result into new coordinates. The paper is self-contained through its replication package, inter-rater reliability reporting, and direct respondent data. The main caveats—sparse chi-square cells, unreported per-item p-values, and possible non-response bias—are statistical robustness concerns, not circularity, and per the hard rules they do not raise the circularity score. I also note an internal reporting inconsistency: §3.1 describes ethics approval and informed consent from 145 participants, while the Ethical Approval section states 'This study does not involve human participants or animals.' That is an integrity/reporting issue, not a circular derivation issue. Overall, the analysis exhibits no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The survey rests on domain assumptions rather than mathematical axioms: self-reports ≈ behavior, sample representativeness, sampling-frame reachability, chi-square validity, phase poolability, and qualitative saturation. No invented entities; two hand-chosen methodological weights (Likert scoring for weighted percentages, alpha threshold). These are reasonable choices for an exploratory survey, but they bound what the divergence percentages can mean.

free parameters (2)
  • Weighted-percentage scoring weights = Critical=4; Very important=3; Moderately important=2; Slightly important=1; Not important=0
    Hand-chosen Likert weights define the Wtd.% columns of Tables 4.1 (selection criteria) and 4.3 (documentation barriers); the headline 'critical %' claims do not depend on these weights, but the secondary rankings (e.g., documentation vs base model disclosure tied at 12.1%) do.
  • Inter-rater reliability threshold = Krippendorff's alpha >= 0.667 (tentative conclusions)
    Adopted from Marzi et al. 2024 to declare coding reliable (observed alphas 0.678-1.000). The threshold choice determines which qualitative themes enter Figure 4.1; a stricter threshold (e.g., 0.80) would have excluded the lowest-agreement themes.
axioms (5)
  • domain assumption Survey self-reports approximate actual behavior
    All four RQs draw on stated practices; authors acknowledge 'responses reflect self-reported practices, which may not precisely correspond to actual behavior' (Section 6).
  • domain assumption Sample representativeness despite 5.72% response rate
    Generalization from 50 producers/95 consumers recruited by cold email to the global PTLM supply chain (Section 3.3, Section 6). Partially mitigated by the 37.9% neutral-consumers argument and reported saturation, but unverifiable.
  • domain assumption Chi-square test conditions hold for the RQ2 contingency tables
    Section 3.4/4.2: cells with 0-5% of n≈41-50 likely produce expected counts below 5 for several of the 20 metadata types, violating a standard chi-square condition; no expected-count diagnostics are reported for the '16 of 20 significant' claim.
  • domain assumption Phase-1 and Phase-2 samples are poolable
    Authors combine probability-based (Phase 1) and purposive (Phase 2) samples and treat them as one cohort (Section 3.3); if the two phases sample different populations (recently active vs any historical contributor), pooled percentages mix two frames.
  • domain assumption Prior mining studies (including the authors' own) provide valid corroboration
    Section 6 argues findings 'align with repository mining studies' (Ajibode et al. 2025, Stalnaker et al. 2025); this treats those studies' measurements as ground truth for cross-method validation.

pith-pipeline@v1.3.0-alltime-deepseek · 21055 in / 22311 out tokens · 198524 ms · 2026-08-01T06:51:34.042764+00:00 · methodology

0 comments
read the original abstract

Pre-trained Language Models (PTLMs) are increasingly reused as dependencies in modern software systems, even though prior work has documented persistent structural problems in PTLM supply chains, such as inconsistent release practices, incomplete metadata, and divergence between Hugging Face and GitHub repositories. One previously unexplored angle on these problems is that misalignment between PTLM producers' release practices and consumers' model reuse needs may explain several of these challenges, yet the human expectations, processes and interactions underneath this misalignment have never been studied. As such, we conducted the first dual-perspective survey of 50 Hugging Face producers and 95 GitHub consumers to examine producer-consumer misalignments across four AI supply chain dimensions: model discovery, documentation practices, lineage tracing, and model governance adoption. We find that, although producers and consumers rely on the same documentation artifacts, they disagree on where critical metadata should be recorded. Even though 27.9% of producers and 31.4% of consumers trace lineage beyond the immediate parent model, producers primarily trace model lineage for provenance and reproducibility, whereas consumers are mainly driven by quality and reliability concerns. Furthermore, producers believe that governance mechanisms that streamline model release would have the greatest positive impact, whereas consumers prioritize documentation and dependency transparency. These findings highlight opportunities to improve model documentation conventions, lineage visibility, and governance support in PTLM supply chains.

Figures

Figures reproduced from arXiv: 2607.21738 by Adekunle Ajibode, Ahmed E. Hassan, Bram Adams, Oussama Ben Sghaier.

Figure 3.1
Figure 3.1. Figure 3.1: Overview of the study methodology. 3.1 Survey Design and Pilot The study received ethics approval from our university prior to data collection. To capture dual perspectives across the AI supply chain, we developed two survey instruments: one for Hugging Face producers with at least one published model (Laufer et al., 2025), and one for GitHub consumers who had reused one or more Hugging Face models in th… view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: Codebook distribution of technical issues, reasons for lineage tracing, lineage tracing methodology, flaw identification, mechanisms that can have positive impact in the AI supply chain, and resistance to governance mechanisms (the last open question in RQ4 is in the replication package). The open question result in [PITH_FULL_IMAGE:figures/full_fig_p008_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Figure 4.2: Primary documentation locations for various model metadata types, comparing producers and consumers. The values represent the percentage distribution of responses for each metadata category [PITH_FULL_IMAGE:figures/full_fig_p009_4_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows that the reasons behind these choices depend on the role. Producers who selected [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 9 linked inside Pith

  1. [4]

    Aspirations and practice of ml model documentation: Moving the needle with nudging and traceability

    Avinash Bhat, Austin Coursey, Grace Hu, Sixian Li, Nadia Nahar, Shurui Zhou, Christian K¨ astner, and Jin LC Guo. Aspirations and practice of ml model documentation: Moving the needle with nudging and traceability. InProceedings of the 2023 CHI Conference on Human Factors in Com- puting Systems, pages 1–17,

    2023

  2. [9]

    From hugging face to github: Tracing license drift in the open-source ai ecosystem.arXiv preprint arXiv:2509.09873,

    James Jewitt, Hao Li, Bram Adams, Gopi Krishnan Rajbahadur, and Ahmed E Hassan. From hugging face to github: Tracing license drift in the open-source ai ecosystem.arXiv preprint arXiv:2509.09873,

    Pith/arXiv arXiv

  3. [10]

    An empirical study of artifacts and security risks in the pre-trained model supply chain

    Wenxin Jiang, Nicholas Synovic, Rohan Sethi, Aryan Indarapu, Matt Hyatt, Taylor R Schorlemmer, George K Thiruvathukal, and James C Davis. An empirical study of artifacts and security risks in the pre-trained model supply chain. InProceedings of the 2022 ACM workshop on software supply chain offensive research and ecosystem defenses, pages 105–114,

    2022

  4. [12]

    Anatomy of a machine learning ecosys- tem: 2 million models on hugging face.arXiv preprint arXiv:2508.06811,

    When Model Release Meets Model Reuse: Producer-Consumer Misalignment in Hugging Face 17 Benjamin Laufer, Hamidah Oderinwale, and Jon Kleinberg. Anatomy of a machine learning ecosys- tem: 2 million models on hugging face.arXiv preprint arXiv:2508.06811,

    Pith/arXiv arXiv

  5. [13]

    Ai transparency atlas: Framework, scoring, and real-time model card evaluation pipeline.arXiv preprint arXiv:2512.12443,

    Akhmadillo Mamirov, Faiaz Azmain, and Hanyu Wang. Ai transparency atlas: Framework, scoring, and real-time model card evaluation pipeline.arXiv preprint arXiv:2512.12443,

    arXiv

  6. [14]

    Building an open aibom standard in the wild.arXiv preprint arXiv:2510.07070,

    Gopi Krishnan Rajbahadur, Keheliya Gallaba, Elyas Rashno, Arthit Suriyawongkul, Karen Bennet, Kate Stewart, and Ahmed E Hassan. Building an open aibom standard in the wild.arXiv preprint arXiv:2510.07070,

    Pith/arXiv arXiv

  7. [15]

    Omnibor: A system for automatic, verifiable artifact resolution across software supply chains.arXiv preprint arXiv:2402.08980,

    Bharathi Seshadri, Yongkui Han, Chris Olson, David Pollak, and Vojislav Tomasevic. Omnibor: A system for automatic, verifiable artifact resolution across software supply chains.arXiv preprint arXiv:2402.08980,

    Pith/arXiv arXiv

  8. [16]

    Naveen Srinivasan, Nathan Naveen, and Neil Naveen. Bomfather: An ebpf-based kernel-level monitor- ing framework for accurate identification of unknown, unused, and dynamically loaded dependencies in modern software supply chains.arXiv preprint arXiv:2503.02097,

    Pith/arXiv arXiv

  9. [17]

    Aibomgen: Generating an ai bill of materials for secure, transparent, and compliant model training

    Wiebe Vandendriessche, Jordi Thijsman, Laurens D’hooge, Bruno Volckaert, and Merlijn Sebrechts. Aibomgen: Generating an ai bill of materials for secure, transparent, and compliant model training. arXiv preprint arXiv:2601.05703,

    arXiv

  10. [18]

    Xinyu Yang, Weixin Liang, and James Zou

    18 Adekunle Ajibode et al. Xinyu Yang, Weixin Liang, and James Zou. Navigating dataset documentations in ai: A large-scale analysis of dataset cards on hugging face.arXiv preprint arXiv:2401.13822,

    Pith/arXiv arXiv

  11. [19]

    Software dependencies 2.0: An empir- ical study of reuse and integration of pre-trained models in open-source projects.arXiv preprint arXiv:2509.06085,

    Jerin Yasmin, Wenxin Jiang, James C Davis, and Yuan Tian. Software dependencies 2.0: An empir- ical study of reuse and integration of pre-trained models in open-source projects.arXiv preprint arXiv:2509.06085,

    arXiv

  12. [20]

    A survey of large language models.arXiv preprint arXiv:2303.18223, 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models.arXiv preprint arXiv:2303.18223, 2023

    Pith/arXiv arXiv 2023

  13. [1995]

    Implementing ai bill of materials (ai bom) with spdx 3.0: A comprehensive guide to creating ai and dataset bill of materials.arXiv preprint arXiv:2504.16743,

    Karen Bennet, Gopi Krishnan Rajbahadur, Arthit Suriyawongkul, and Kate Stewart. Implementing ai bill of materials (ai bom) with spdx 3.0: A comprehensive guide to creating ai and dataset bill of materials.arXiv preprint arXiv:2504.16743,

    arXiv

  14. [2006]

    Joel Casta˜ no, Rafael Caba˜ nas, Antonio Salmer´ on, David Lo, and Silverio Mart ´ ınez-Fern´ andez

    16 Adekunle Ajibode et al. Joel Casta˜ no, Rafael Caba˜ nas, Antonio Salmer´ on, David Lo, and Silverio Mart ´ ınez-Fern´ andez. How do machine learning models change?arXiv preprint arXiv:2411.09645,

    arXiv

  15. [2021]

    Modelgo: A practical tool for machine learning license analysis

    Moming Duan, Qinbin Li, and Bingsheng He. Modelgo: A practical tool for machine learning license analysis. InProceedings of the ACM Web Conference 2024, pages 1158–1169,

    2024

  16. [2022]

    Naming practices of pre-trained models in hugging face.arXiv preprint arXiv:2310.01642, 2023a

    Wenxin Jiang, Chingwo Cheung, Mingyu Kim, Heesoo Kim, George K Thiruvathukal, and James C Davis. Naming practices of pre-trained models in hugging face.arXiv preprint arXiv:2310.01642, 2023a. Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K Thiruvathukal, and James C Davis. An empirical study of pre-t...

    Pith/arXiv arXiv

  17. [2023]

    On the opportu- nities and risks of foundation models.arXiv preprint arXiv:2108.07258,

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models.arXiv preprint arXiv:2108.07258,

    Pith/arXiv arXiv

  18. [2024]

    Empirical char- acterization of logging smells in machine learning code.arXiv preprint arXiv:2603.23769,

    Patrick Loic Foalem, Leuson Da Silva, Foutse Khomh, Heng Li, and Ettore Merlo. Empirical char- acterization of logging smells in machine learning code.arXiv preprint arXiv:2603.23769,

    arXiv

  19. [2025]

    On the synchronization between hugging face pre-trained language models and their upstream github repository.arXiv preprint arXiv:2508.10157,

    Adekunle Ajibode, Abdul Ali Bangash, Bram Adams, and Ahmed E Hassan. On the synchronization between hugging face pre-trained language models and their upstream github repository.arXiv preprint arXiv:2508.10157,

    arXiv

  20. [2026]

    Version v1

    URLhttps://zenodo.org/records/20647278. Version v1. Sebastian Baltes and Paul Ralph. Sampling in software engineering research: A critical review and guidelines.Empirical Software Engineering, 27(4):94,

    arXiv