Pith. sign in

REVIEW 2 major objections 2 minor 54 references

Large language models can construct partial orderings of software licenses based on permissiveness through pairwise comparisons.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 05:12 UTC pith:MHT3XX6U

load-bearing objection LLM pairwise license comparisons are a new angle but the abstract shows no validation against experts or known legal cases. the 2 major comments →

arxiv 2606.31032 v1 pith:MHT3XX6U submitted 2026-06-30 cs.SE cs.CY

Partially ordering software licenses

Hamidah Oderinwale , David Atkinson , Rachel Hong , Art Abal , Ben Laufer This is my paper
classification cs.SE cs.CY
keywords software licensespartial orderingpermissivenesslarge language modelsopen sourcelicense taxonomieslegal implications
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to show that large language models offer a way to compare software licenses at scale by making pairwise judgments of permissiveness and by using existing taxonomies of license terms. This addresses the problem that there is no systematic method to determine when one license is more permissive than another or when they are incomparable. A sympathetic reader would care because platforms hosting open-source software need to manage license compatibility as the number of licenses grows. If successful, the approach traces restrictiveness across licenses and identifies shared provisions that influence selection. It also recovers attributes that mark stricter licenses, carrying legal implications for the ecosystem.

Core claim

Using large language models, licenses can be compared pairwise to build a partial ordering based on permissiveness, and taxonomies can be used to understand license selection as combinations of shared provisions. This recovers interpretable attributes that correspond to stricter licenses.

What carries the argument

Pairwise comparisons by large language models to establish relative permissiveness, together with mappings onto existing license taxonomies.

Load-bearing premise

Large language models generate comparisons of license terms that are consistent and align with expert legal judgment instead of reflecting prompt sensitivity or training data patterns.

What would settle it

If copyright lawyers systematically disagree with the LLM pairwise rankings on a representative set of licenses regarding which imposes more restrictions on reuse and modification.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • License relationships become traceable at scale rather than remaining unstructured.
  • Platforms can identify when licenses are incomparable rather than assuming total orders.
  • Stricter license attributes become detectable through the recovered features.
  • License selection can be analyzed as choices among shared provisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach could be tested on emerging licenses to see if the ordering holds.
  • Legal experts might use the attributes to flag potential conflicts in license combinations.
  • Extending the method to other legal documents could reveal similar partial orders in contract terms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces LLM-based methods for comparing software licenses at scale. The first constructs a partial order on permissiveness via pairwise judgments; the second projects licenses onto existing taxonomies to identify combinations of shared provisions. The analysis claims to recover interpretable attributes associated with stricter licenses and discusses implications for the open-source ecosystem on platforms such as GitHub and Hugging Face.

Significance. If the LLM judgments can be shown to align with expert legal reasoning, the approach would supply a scalable, reproducible technique for license analysis that is currently absent from the literature. The work is novel in applying LLMs to this domain and could support practical tooling for license compatibility checking, but its contribution is limited by the absence of validation evidence.

major comments (2)
  1. [Methods] Methods section: The pairwise prompting procedure for constructing the partial order is described without any reported validation against practicing licensing counsel, inter-annotator agreement metrics, or calibration against established legal distinctions (e.g., strong copyleft in GPL-family licenses versus permissive MIT/BSD terms). This validation is load-bearing for the central claim that the recovered ordering reflects legal permissiveness rather than training-data artifacts or prompt sensitivity.
  2. [Results] Results section: No details are supplied on the prompting strategy (including temperature, few-shot examples, or consistency checks), the procedure for aggregating pairwise judgments into a partial order, or robustness under prompt paraphrases. Without these, the reported interpretable attributes cannot be assessed for stability or legal fidelity.
minor comments (2)
  1. [Abstract] Abstract: The number of licenses examined and the specific LLMs employed are not stated, making it difficult to gauge the scale of the study.
  2. [Methods] Notation: The manuscript should define how incomparability is operationalized in the partial order (e.g., when two licenses receive conflicting pairwise judgments).

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback. We address the two major comments point by point below, with planned revisions where feasible.

read point-by-point responses

  1. Referee: [Methods] Methods section: The pairwise prompting procedure for constructing the partial order is described without any reported validation against practicing licensing counsel, inter-annotator agreement metrics, or calibration against established legal distinctions (e.g., strong copyleft in GPL-family licenses versus permissive MIT/BSD terms). This validation is load-bearing for the central claim that the recovered ordering reflects legal permissiveness rather than training-data artifacts or prompt sensitivity.

    Authors: We agree that external validation against legal experts would strengthen claims of legal fidelity. The manuscript positions LLMs as a scalable proxy rather than a replacement for counsel; however, we will revise to add inter-annotator agreement via repeated runs with varied seeds and a calibration subsection comparing known distinctions (GPL-family vs. MIT/BSD). Full engagement with practicing licensing counsel lies outside the scope of this work and would require a separate study. revision: partial

  2. Referee: [Results] Results section: No details are supplied on the prompting strategy (including temperature, few-shot examples, or consistency checks), the procedure for aggregating pairwise judgments into a partial order, or robustness under prompt paraphrases. Without these, the reported interpretable attributes cannot be assessed for stability or legal fidelity.

    Authors: We accept this criticism and will expand the Methods section in revision. The updated text will report temperature=0, the complete prompt templates (with any few-shot examples), the aggregation procedure (directed graph followed by transitive reduction to obtain the partial order), and new robustness results under prompt paraphrases demonstrating stability of the recovered attributes. revision: yes

standing simulated objections not resolved
  • Formal validation against practicing licensing counsel

Circularity Check

0 steps flagged

No circularity: LLM pairwise comparisons and taxonomy projection are independent of target ordering

full rationale

The paper's method applies LLMs to generate pairwise permissiveness judgments and projects onto existing taxonomies to recover attributes. No equations, fitted parameters, or self-definitional reductions appear. The partial order is constructed from model outputs rather than presupposing the result; no self-citation chain justifies a uniqueness theorem or ansatz. The derivation remains self-contained against external benchmarks (legal taxonomies) and does not rename known patterns or smuggle inputs as predictions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are described.

pith-pipeline@v0.9.1-grok · 5736 in / 925 out tokens · 21294 ms · 2026-07-01T05:12:56.320197+00:00 · methodology

0 comments
read the original abstract

Licenses are legal instruments that inventors may use to protect the technologies they build and regulate how they are used -- however, the nature of their authorship and selection means that how they are interpreted, chosen, and enforced is largely unstructured. In practice, this makes it difficult to compare licenses at scale -- when is one license considered more permissive than the other, and when are their terms incomparable to each other? Currently, there is a growing list of licenses that are introduced and used, but there is no systematic way to study their relationships. This matters for platforms such as Hugging Face, GitHub, and the Python Package Index, where developers publish or build upon technologies that each have their own licenses. Using large language models (LLMs), we introduce methods for comparing licenses at scale: first, in a pairwise fashion to construct a partial ordering based on permissiveness, and second, by drawing on existing taxonomies of software licenses. The former allows us to trace restrictiveness, and the latter allows us to understand license selection as a combination of shared provisions. Our analysis recovers certain interpretable attributes that correspond to stricter licenses, with legal implications for the open-source ecosystem.

Figures

Figures reproduced from arXiv: 2606.31032 by Art Abal, Ben Laufer, David Atkinson, Hamidah Oderinwale, Rachel Hong.

Figure 1
Figure 1. Figure 1: How often features appear across licenses. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Barbell plot showing how Nordlander features change in directed pairs, where [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Lattice diagram of sequential feature selection over license space. Three features [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example software library dependency chains in PyPI. Each node is a package [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Severity of license inconsistencies (violations of permissive ordering) by software [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Top 6 most common license inconsistencies over time across software packages, [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The agreement between models (Sonnet and DeepSeek) and prompt iterations for [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cohen’s κ analysis for features between Sonnet 4.6 and DeepSeek. For all extrac￾tions, temperature was set to 0. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Breakdown of incomparability rationales. Haiku 4.5 is used for canonicalization [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Features plotted based on logistic regression on permissiveness. [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Upset plot depicting differences in ordered license pairs grouped by the number [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 54 canonical work pages

  1. [1]

    Harvard Law Review , volume=

    Property as the law of things , author=. Harvard Law Review , volume=

    work page

  2. [2]

    arXiv preprint arXiv:2504.20185 , year=

    The Foundation Model Transparency Index , author=. arXiv preprint arXiv:2504.20185 , year=

    work page arXiv

  3. [3]

    arXiv preprint arXiv:2401.xxxxx , year=

    Data provenance and the machine learning supply chain , author=. arXiv preprint arXiv:2401.xxxxx , year=

    work page

  4. [4]

    2022 , month=

    Ensuring Free, Immediate, and Equitable Access to Federally Funded Research , author=. 2022 , month=

    work page 2022

  5. [5]

    2023 , note=

    Model and dataset documentation on Hugging Face , author=. 2023 , note=

    work page 2023

  6. [6]

    2024 , howpublished=

    Model deprecation policy , author=. 2024 , howpublished=

    work page 2024

  7. [7]

    arXiv preprint arXiv:2403.xxxxx , year=

    Model ChangeLists: Documenting API model performance changes , author=. arXiv preprint arXiv:2403.xxxxx , year=

    work page

  8. [8]

    2024 , howpublished=

    News Corp and OpenAI partnership agreement , author=. 2024 , howpublished=

    work page 2024

  9. [9]

    2024 , note=

    The New York Times licensing and litigation regarding AI training , author=. 2024 , note=

    work page 2024

  10. [10]

    Innovation Policy and the Economy , volume=

    Navigating the patent thicket: Cross licenses, patent pools, and standard setting , author=. Innovation Policy and the Economy , volume=. 2001 , publisher=

    work page 2001

  11. [11]

    Common Crawl , author=

    work page

  12. [12]

    Schuhmann, Christoph and others , journal=

    work page

  13. [13]

    Gao, Leo and Biderman, Stella and Black, Sid and others , journal=. The

    work page

  14. [14]

    Books3 dataset , author=

    work page

  15. [15]

    Stability AI complaint , author=

    Getty Images v. Stability AI complaint , author=. 2023 , howpublished=

    work page 2023

  16. [16]

    2023 , note=

    Authors Guild class action lawsuits against AI companies , author=. 2023 , note=

    work page 2023

  17. [17]

    2023 , howpublished =

    Llama 2 Community License Agreement , author =. 2023 , howpublished =

    work page 2023

  18. [18]

    2004 , month =

    Nordlander, Erik and Loreto, Daniel and Oliner, Adam and Woo, Ram , title =. 2004 , month =

    work page 2004

  19. [19]

    The Rise of Open Source Licensing:

    V. The Rise of Open Source Licensing:. 2005 , address =

    work page 2005

  20. [20]

    A dataset showing a century of evolution in the complexity of the United States legal Code

    Jeong, Dawoon and Holehouse, James and Yoon, Jisung and Kempes, Christopher P and West, Geoffrey B and Youn, Hyejin. A dataset showing a century of evolution in the complexity of the United States legal Code. Sci. Data

    work page

  21. [21]

    2025 , eprint=

    Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face , author=. 2025 , eprint=

    work page 2025

  22. [22]

    Lewis & Clark Law Review , year =

    Madiha Zahrah Choksi and James Grimmelmann , title =. Lewis & Clark Law Review , year =

    work page

  23. [23]

    2025 , eprint=

    Open Problems in Technical AI Governance , author=. 2025 , eprint=

    work page 2025

  24. [24]

    Cen, Isabella Struckman, Andrew Ilyas, Luis Videgaray, and Aleksander Mądry

    Hopkins, Aspen and Cen, Sarah H. and Struckman, Isabella and Ilyas, Andrew and Videgaray, Luis and Mądry, Aleksander , year=2025, pages=. AI Supply Chains: An Emerging Ecosystem of AI Actors, Products, and Services , volume=. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , publisher=. doi:10.1609/aies.v8i2.36628 , number=

    work page doi:10.1609/aies.v8i2.36628 2025

  25. [25]

    2023 , eprint=

    The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI , author=. 2023 , eprint=

    work page 2023

  26. [26]

    and Chen, Xi and Zhang, Jingxuan , year =

    Bayar, Ozgur and Chemmanur, Thomas J. and Chen, Xi and Zhang, Jingxuan , year =. The Economics of Patent Licensing: Theory and Evidence on the Determinants and Consequences of Patent Licensing Transactions , note =

    work page

  27. [27]

    2026 , eprint=

    Permissive-Washing in the Open AI Supply Chain: A Large-Scale Audit of License Integrity , author=. 2026 , eprint=

    work page 2026

  28. [28]

    and Eisenberg, Rebecca S

    Heller, Michael A. and Eisenberg, Rebecca S. , title =. Science , year =. doi:10.1126/science.280.5364.698 , issn =

    work page doi:10.1126/science.280.5364.698

  29. [29]

    American Economic Review , year =

    Ostrom, Elinor , title =. American Economic Review , year =

    work page

  30. [30]

    ArXiv , year=

    Examining Software Developers' Needs for Privacy Enforcing Techniques: A survey , author=. ArXiv , year=

    work page

  31. [31]

    Myers, Christopher R. , year=. Software systems as complex networks: Structure, function, and evolvability of software collaboration graphs , volume=. Physical Review E , publisher=. doi:10.1103/physreve.68.046116 , number=

    work page doi:10.1103/physreve.68.046116

  32. [32]

    2017 , howpublished =

    Conor Mayo-Wilson , title =. 2017 , howpublished =

    work page 2017

  33. [33]

    Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

    Snow, Rion and O ' Connor, Brendan and Jurafsky, Daniel and Ng, Andrew. Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 2008

    work page 2008

  34. [34]

    A Mathematical Theory of Communication , url =

    Shannon, Claude Elwood , biburl =. A Mathematical Theory of Communication , url =. The Bell System Technical Journal , keywords =

    work page

  35. [35]

    2024 , month = dec, number =

    System Package Data Exchange (. 2024 , month = dec, number =

    work page 2024

  36. [36]

    2025 , eprint=

    A Survey on LLM-as-a-Judge , author=. 2025 , eprint=

    work page 2025

  37. [37]

    1994 , publisher =

    Brand, Stewart , title =. 1994 , publisher =

    work page 1994

  38. [38]

    Donald Arseneau , howpublished =

    work page

  39. [39]

    1999 , howpublished =

    Houston, Gary , title =. 1999 , howpublished =

    work page 1999

  40. [40]

    Artistic License 1.0 (Perl) , howpublished =

    work page

  41. [41]

    2024 , eprint=

    The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence , author=. 2024 , eprint=

    work page 2024

  42. [42]

    Yale Law Journal , volume =

    Wesley Newcomb Hohfeld , title =. Yale Law Journal , volume =

    work page

  43. [43]

    , title =

    Kuhn, Thomas S. , title =. 1962 , publisher =

    work page 1962

  44. [44]

    IEEE Transactions on Visualization and Computer Graphics (InfoVis) , doi =

    UpSet: Visualization of Intersecting Sets , author =. IEEE Transactions on Visualization and Computer Graphics (InfoVis) , doi =

    work page

  45. [45]

    Proceedings of the 21st International Conference on Mining Software Repositories , pages =

    Wu, Jiaqi and Bao, Lingfeng and Yang, Xiaohu and Xia, Xin and Hu, Xing , title =. Proceedings of the 21st International Conference on Mining Software Repositories , pages =. 2024 , isbn =. doi:10.1145/3643991.3644900 , abstract =

    work page doi:10.1145/3643991.3644900 2024

  46. [46]

    and Charalambous, Georgia , journal=

    Kapitsaki, Georgia M. and Charalambous, Georgia , journal=. Modeling and Recommending Open Source Licenses with findOSSLicense , year=

    work page

  47. [47]

    Terry , journal =

    Ralph Allan Bradley and Milton E. Terry , journal =. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , urldate =

    work page

  48. [48]

    Arrow , publisher =

    Kenneth J. Arrow , publisher =. Social Choice and Individual Values , urldate =

    work page

  49. [49]

    Forty-second International Conference on Machine Learning Position Paper Track , year=

    Position: Current Model Licensing Practices are Dragging Us into a Quagmire of Legal Noncompliance , author=. Forty-second International Conference on Machine Learning Position Paper Track , year=

    work page

  50. [50]

    2004 , url=

    Understanding open source and free software licensing - guide to navigation licensing issues in existing and new software , author=. 2004 , url=

    work page 2004

  51. [51]

    Reddy , journal =

    Hersb R. Reddy , journal =. Jacobsen v. Katzer: The Federal Circuit Weighs in on the Enforceability of Free and Open Source Software Licenses , urldate =

    work page

  52. [52]

    2025 , note =

    Knowing Machines , title =. 2025 , note =

    work page 2025

  53. [53]

    , note =

    Google , title =. , note =

    work page

  54. [54]

    2026 , note =

    Rohit Gupta , title =. 2026 , note =

    work page 2026