Pith. sign in

REVIEW 5 major objections 4 minor 97 references

From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A composite labeling function built on LLM-extracted concepts turns noisy patent citations into silver-standard SDG labels that beat existing baselines.

desk verdict A plausible weak-supervision pipeline for patent-SDG labels, but the evidence that the new labels are trustworthy rests on a p-dependent, unquantified modularity test, and the internal recall is coupled to the calibration signal. read the letter →

arxiv 2509.09303 v1 pith:EFIRRSZ4 submitted 2025-09-11 cs.CL

Grazia Sveva Ascione , Nicolò Tamagnone This is my paper · ORCID
classification cs.CL
keywords weaksupervisionlabelingfunctionsSDGclassificationpatentlargelanguagemodelssemanticsimilaritynetworkmodularitynon-patentliterature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a weak-supervision labeling function can generate trustworthy patent-to-SDG labels without manual annotation or a gold standard. The function uses a large language model to extract 'functions, solutions, applications' from both patents and SDG-tagged scientific papers, measures semantic similarity with embeddings, fuses rankings with Reciprocal Rank Fusion, and calibrates its thresholds with a positive-only loss that does not penalize discovering new SDG associations. The resulting soft 17-dimensional relevance vectors recover known NPL-derived associations better than BM25, fine-tuned transformers, and zero-shot LLMs, and they create more coherent communities in citation, inventor, and applicant networks than CPC or NPL labels. If right, this gives a scalable way to build training data for SDG classification over the global patent corpus.

What carries the argument

The composite labeling function λ(p) is defined by three stages. First, an LLM (gpt-4.1-mini) is prompted to extract functions, solutions, and applications from each patent and SDG-tagged paper. Second, per-category cosine similarities are computed with learned thresholds (τ_func, τ_sol, τ_app), and the filtered rankings are fused via Reciprocal Rank Fusion: RRF(d) = Σ_i 1/(k + rank_i(d)). The top-N aligned documents then yield a normalized 17-dimensional SDG frequency vector. Third, a positive-only MSE loss (Eq. 3) tunes {τ_i} and N against known NPL citations, so that new SDG assignments are not penalized. This loss is central because it operationalizes the weak-supervision assumption of l

What would settle it

Take a random sample of, say, 500 patents from the silver dataset, have human experts independently annotate SDG relevance, and measure agreement (e.g., rank correlation) with the silver vectors. If agreement is near chance, the labels are not trustworthy. Alternatively, permute the SDG tags of the retrieved scientific papers and rerun the pipeline: if the modularity advantage persists under permutation, the signal comes from something other than the tags.

Watch

Extended reading notes

Core claim

The central claim is that the composite labeling function λ(p) produces silver-standard soft SDG labels that are high enough quality to train downstream multi-label regression models. Concretely, the silver labels achieve the highest recall against the NPL-derived reference (macro 0.711, micro 0.902) and the highest overlapping modularity across all three patent networks (citation 1.05, inventor 0.98, applicant 0.68), exceeding both CPC and NPL baselines. The method also expands coverage: 150,946 SDG labels across 32,895 patents versus 43,956 NPL labels, raising the average number of SDGs per patent from 1.34 to 4.59. The authors interpret this as evidence that semantically grounded labels c

Load-bearing premise

The load-bearing premise is that the query-based SDG tags on the scientific corpus are correct; if those tags are wrong or systematically biased, every silver label, the recall evaluation, and the modularity validation inherit that error.

Editorial extensions

If this is right

  • The released silver-standard dataset (32,895 patents, 150,946 SDG labels) can directly serve as training data for multi-label SDG classifiers without manual annotation.
  • A patent's SDG profile can be predicted from title and abstract alone; the paper shows a multi-output regression model with PAT-SPECTER embeddings reaches MSE 0.0187, indicating the silver signal is learnable.
  • Because the positive-only calibration does not penalize new SDG matches, the method can label patents with no scientific citations at all and still produce coherent assignments.
  • The higher modularity of silver labels in citation, inventor, and applicant networks indicates that SDG alignment captures thematic and organizational structure that CPC technology classes miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same weak-supervision recipe could be transferred to other noisy bibliometric links (e.g., patent-to-patent citations, grant acknowledgements) to generate soft labels for any policy-relevant taxonomy, not just SDGs.
  • The SDG queries used to tag the scientific corpus bias toward English-language journal articles, so the silver dataset likely underrepresents SDG-relevant innovation from non-English or non-journal literature; a multilingual extension would test that bias.
  • The modularity advantage at low power-parameter values suggests that low-probability secondary SDG assignments carry real structural signal; a human-subject study could check whether experts also find those weak labels plausible.
  • Sensitivity to the fixed ontology (functions/solutions/applications) could be probed by swapping in an alternative patent ontology and re-running the pipeline to see whether recall and modularity degrade.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a weak-supervision pipeline for patent-to-SDG classification. It starts from citations that patents make to SDG-tagged scientific publications (NPL citations) and builds a composite labeling function λ(p) that: (i) uses an LLM to extract functions, solutions, and applications from patent and paper texts; (ii) computes category-wise cosine similarities and fuses them with Reciprocal Rank Fusion; and (iii) converts the top-N retrieved papers' SDG tags into a soft 17-dimensional relevance vector (Eq. 2). Hyperparameters are tuned on 40% of the data using a positive-only MSE loss (Eq. 3) against the NPL-derived SDG distribution. The remaining 60% is used to produce a silver-standard dataset of 32,895 patents, which is then used to train multi-output regression models. The authors validate the silver labels internally by recall against NPL-derived SDG labels (Table 4) and externally by overlapping modularity in citation, co-inventor, and co-applicant networks (Table 7, Figure 3). They report that the silver labels outperform baselines in both evaluations and are learnable in the downstream regression setting.

Significance. If the validation concerns are resolved, this would be a useful contribution: it addresses a real annotation bottleneck, proposes a scalable and interpretable LF, and brings network homophily as an external check on labels. The paper is well situated in the weak-supervision and patent-SDG literatures, and it includes a clear ablation study. The authors are candid about several limitations in Section 6. However, the central claim that the silver labels are 'trustworthy' is not established by the current evidence: the reported internal score does not correspond to the final dataset, the internal evaluation is calibrated against the same NPL signal it measures, and the external modularity result is p-dependent and lacks a density-matched null. The paper does not state plans to release the dataset or code, which further limits reproducibility for a claimed public training resource.

major comments (5)
  1. [§4.1, Table 4 and ablation] The row labeled 'Silver' in Table 4 reports the full three-concept LF (macro recall 0.711, micro recall 0.902). However, the ablation in the same section shows that 'Solution & Function' achieves higher recall (0.725/0.910), and the text explicitly states 'we remove this dimension in the final construction of the silver dataset.' Thus the headline internal recall is not for the labeling function used to create the released/trained-on dataset. Please report the actual final LF's numbers and re-run the external validation on that final version.
  2. [§3.2.1, Eq. (3); §4.1] The LF hyperparameters {τ_func, τ_sol, τ_app, top-N} are optimized on 40% of the data to match the NPL-derived SDG distribution, and Table 4 then measures recall against the same NPL-derived SDG labels. Even if the evaluation uses the held-out 60%, the metric and the calibration objective are the same noisy signal, and the top-N=30 used for all baselines is itself the optimized LF hyperparameter. This does not demonstrate that newly added SDG labels are valid; it shows that the method can recover the calibration signal. The paper acknowledges the need for external validation at the start of §4.2, but the abstract and conclusion should not rest on Table 4 as evidence of trustworthiness.
  3. [§4.2, Table 7, Figure 3] Table 7 reports overlapping modularity only at p=0.35, but Figure 3 shows that the ordering is p-dependent: near p≈0.8–1.0, NPL and the transformer baselines match or outperform Silver, and Silver's advantage appears only for low p. This is exactly the regime where weak membership weights are amplified, and the Silver labels are much denser than the NPL labels (4.59 vs 1.34 labels per patent, §5.1). Without a density-matched permutation null that preserves per-node label counts and SDG prevalences, and without a p-sweep with significance estimates, the comparison (e.g., 1.05 vs 1.00 on citations) is confounded by label density/entropy. The paper's own text says that near p=1 'npl and the baselines outperform silver,' so the external-validity conclusion is not established.
  4. [§3.1, Eq. (2), Appendix A.1] The entire pipeline treats the Elsevier SDG query tags as ground truth: the silver label is the frequency of these tags among top-N retrieved papers, the internal recall is measured against citations to these tags, and the external modularity uses them as membership weights. There is no independent check of the SDG tags themselves, and the query-splitting procedure in Appendix A.1 is a possible source of subtle errors. Please provide a sample-based validation against expert annotation or an alternative SDG mapping (e.g., manually labeled papers or a different SDG classifier), and report sensitivity to the query-splitting implementation. Without this, a systematic bias in the Elsevier replication propagates into every label and both validations.
  5. [§3.1, Table 3] The dataset description is internally inconsistent. The text reports 55,008 patents citing 21,517 of the top-cited papers and then says 36,942 unique papers are mapped to at least one SDG; Table 3 reports 41,736 SDG-labeled publications and 73,266 patents and calls them unique, while the table's totals are in fact association counts. Since the silver-set construction and the 40/60 split depend on this base population, please reconcile the numbers and clearly distinguish unique patents/publications from patent-goal and paper-goal counts.
minor comments (4)
  1. [§4.1, Table 4 note] Please specify the binarization rule used to convert soft SDG vectors into the predicted labels counted in recall. It is unclear whether a nonzero score in the top-30 output is counted as a prediction, and whether this rule is identical for the LF and all baselines.
  2. [§4.2, Table 7] State how the NPL and CPC membership vectors for modularity are constructed. If NPL is binarized while Silver is soft, or if CPC membership uses a different normalization, the comparison at a fixed p is not apples-to-apples.
  3. [§5.2, Table 8] The learnability evaluation would benefit from a trivial baseline, such as a constant predictor or an untrained embedding-plus-linear model. The MSE values alone do not show that the silver signal is learnable beyond the model's capacity to memorize labels.
  4. [Throughout] Typos and minor wording issues: 'avergage' (Section 5.1), 'diagolals' (Figure 5 caption), 'SGG9' (Section 3.1), 'indirected network' (Section 4.2), 'The point is corroborated Narin et al.' (Section 2.2, missing 'by'). A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the internal evaluation uses a held-out split and the external network modularity test is independent of label construction.

full rationale

The derivation chain is not circular. Silver labels are produced by Eq. (2) as the normalized SDG frequency among the top-N retrieved SDG-tagged papers; this is a retrieval output, not a fitted parameter. The only fitted quantities are the three similarity thresholds and the top-N cutoff, which are tuned on a 40% stratified validation split by minimizing the positive-only loss in Eq. (3) against NPL-derived SDG distributions. The internal evaluation (Table 4) is run on the held-out 60%, as stated in the abstract: 'internal validation against held-out NPL-based labels.' The recall figures therefore measure generalization to patents whose NPL labels were not used in calibration; this is a standard model-selection split, not a fitted input renamed as a prediction. Eq. (2) and Eq. (3) are not equal by construction: one is the predicted SDG distribution from retrieved documents, the other is the target SDG distribution from cited papers, and they are connected only through the optimization objective. The external modularity evaluation is also not circular: the three networks (citations, co-inventors, co-applicants) are exogenous to the label-generation pipeline, and the labels are not derived from the network structure. Concerns about the choice p=0.35, label-density confounds, and the absence of a permutation null are threats to the strength of the external conclusion, but they are correctness/validity concerns rather than circularity. The only self-citation (Ascione 2023) appears in the literature review and is not load-bearing for any central claim. Limitations are explicitly acknowledged in Section 6 (e.g., selection bias, heuristic homophily proxies, LLM sensitivity), and the paper does not hide its reliance on the Elsevier SDG query replication. Overall, the central claims are supported by a held-out internal benchmark and an independent, if imperfect, network-based validation.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central contribution (silver SDG labels) rests on the external SDG tags for publications, the citation-based positive anchors, the ontology categories, and the embedding similarity. The four fitted thresholds and the chosen p parameter are the main ad hoc degrees of freedom; none have independent evidence outside the paper's own evaluation.

free parameters (7)
  • Semantic similarity threshold for functions (τ_function) = 0.260
    Optimized on the 40% validation split via Optuna to maximize alignment with NPL-derived SDG distributions (Sec 3.2.1).
  • Semantic similarity threshold for solutions (τ_solution) = 0.165
    Optimized on the validation split via Optuna (Sec 3.2.1).
  • Semantic similarity threshold for applications (τ_application) = 0.873
    Optimized on the validation split via Optuna (Sec 3.2.1).
  • Top-N cutoff for retrieved SDG documents = 30
    Optimized on the validation split; defines how many SDG papers contribute to each patent's SDG frequency vector (Sec 3.2.1).
  • Modularity power parameter p = 0.35
    Hand-chosen in the overlapping modularity computation to emphasize weak overlaps; the paper shows that silver labels overtake baselines specifically in the low-p regime (Sec 4.2, Figure 3).
  • RRF dampening constant k
    Appears in Eq 1 but its value is not reported in the paper; affects the fused ranking and thus the SDG vectors.
  • Top-cited paper cutoff per SDG = 20,000
    Each SDG contributes the 20,000 most-cited papers to the citation pool; this data filter shapes the Patent-SDG Dataset and is a design choice, not fitted.
assumptions (6)
  • domain assumption Elsevier SDG Mapping Initiative queries correctly identify scientific publications relevant to each SDG, and the authors' segmentation of these queries preserves logical equivalence.
    All paper-side labels come from re-implementing these queries via the Scopus API (Sec 3.1); any tagging error propagates to the silver labels and all validations.
  • domain assumption NPL citations from patents to SDG-tagged papers are informative but noisy and incomplete anchors for SDG relevance.
    The weak supervision setup uses these citations as positive-only targets for calibration (Eq 3) and as the reference for internal evaluation (Sec 4.1).
  • ad hoc to paper The functions-solutions-applications ontology of Zhai et al. (2022) provides a sufficient shared semantic space for aligning patents and papers.
    The LF is built around these three concept categories without an independent test that they capture all relevant patent-paper semantic overlap.
  • domain assumption Cosine similarity between PAECTER embeddings of the LLM-extracted concepts captures semantic relatedness between patent and paper concepts.
    The retrieval ranking and all derived SDG vectors rely on this embedding similarity (Sec 3.2.1).
  • domain assumption Overlapping modularity (Nicosia et al.) with a power affinity function is a valid proxy for label quality in the absence of ground truth.
    The external validation (Sec 4.2) equates higher modularity of SDG labels across citation, inventor, and applicant edges with higher label quality.
  • ad hoc to paper The positive-only MSE loss (Eq 3) implements the intended property of not penalizing discovery of new SDG associations.
    The paper claims this property, but because the predicted vector is normalized over the top-N retrieved papers (Eq 2), assigning mass to a new SDG necessarily reduces the predicted probability of cited SDGs and increases the squared error on those positives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models." pith.science (2026). https://pith.science/paper/EFIRRSZ4

@misc{pith2026250909303,
  author       = {Pith},
  title        = {Pith review of: From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFIRRSZ4}},
  note         = {Machine review of arXiv:2509.09303}
}
read the original abstract

Classifying patents by their relevance to the UN Sustainable Development Goals (SDGs) is crucial for tracking how innovation addresses global challenges. However, the absence of a large, labeled dataset limits the use of supervised learning. Existing methods, such as keyword searches, transfer learning, and citation-based heuristics, lack scalability and generalizability. This paper frames patent-to-SDG classification as a weak supervision problem, using citations from patents to SDG-tagged scientific publications (NPL citations) as a noisy initial signal. To address its sparsity and noise, we develop a composite labeling function (LF) that uses large language models (LLMs) to extract structured concepts, namely functions, solutions, and applications, from patents and SDG papers based on a patent ontology. Cross-domain similarity scores are computed and combined using a rank-based retrieval approach. The LF is calibrated via a custom positive-only loss that aligns with known NPL-SDG links without penalizing discovery of new SDG associations. The result is a silver-standard, soft multi-label dataset mapping patents to SDGs, enabling the training of effective multi-label regression models. We validate our approach through two complementary strategies: (1) internal validation against held-out NPL-based labels, where our method outperforms several baselines including transformer-based models, and zero-shot LLM; and (2) external validation using network modularity in patent citation, co-inventor, and co-applicant graphs, where our labels reveal greater thematic, cognitive, and organizational coherence than traditional technological classifications. These results show that weak supervision and semantic alignment can enhance SDG classification at scale.

Figures

Figures reproduced from arXiv: 2509.09303 by the authors.

Figure 1
Figure 1. Data integration pipeline for constructing the [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Labeling function Notes: The Figure illustrates the labeling function pipeline. Patent and SDG documents from the Patent-SDG Dataset are processed using LLMs to extract ontological concepts (functions, solutions, applications). Semantic similarities are computed between patent and SDG concepts, and low-similarity pairs (below τsem) are filtered out. Concepts are ranked and fused via Reciprocal Rank Fusion (RRF). Wea… view at source ↗
Figure 3
Figure 3. Overlapping modularity Q(p) (directed Nicosia; F(a, b) = (a p )(b p )) across labeling methods. Panels: (a) citation graph, (b) inventor graph, (c) applicant graph. By analyzing modularity across these three networks, we are able to evaluate the extent to which labeling schemes align with different forms of structural homophily: thematic proximity (citations), cogni￾tive proximity (inventors), and organizational pro… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of SDG label distributions between the NPL signal (red) and the silver signal [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]
Figure 5
Figure 5. Figure 5: Row-wise normalized SDG co-occurrence matrices for the NPL signal (left) and the silver [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]
Figure 6
Figure 6. Figure 6: Correlation between CPC3 classes and SDG targets using two SDG labeling signals. [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]
Figure 7
Figure 7. Figure 7: SDG publications trend (2015–2022) Notes: The Figure displays the number of SDG-tagged papers published annually between 2005 and 2022 by the Elsevier project. If a paper is classified under multiple SDGs, it is counted once per SDG. Only English-language journal artic…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 24 linked inside Pith

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    https://www.wipo.int/edocs/pubdocs/en/wipo-pub-rn2024-18-en-mapping-innovations.pdf

    author WIPO , title Mapping Innovations: Patents and the Sustainable Development Goals , type Report number WIPO-PUB-RN2024-18 , World Intellectual Property Organization, year 2024 . https://www.wipo.int/edocs/pubdocs/en/wipo-pub-rn2024-18-en-mapping-innovations.pdf

    2024

  4. [4]

    author F. Mazzi , title Patents, social welfare, and sustainability: considerations on the role of the patent system in incentivising the un sustainable development goals , journal A Research Agenda for Patent Law’, Edward Elgar, Forthcoming ( year 2024 )

    2024

  5. [5]

    author J. W. Van der Waal , author T. Thijssens , author K. Maas , title The innovative contribution of multinational enterprises to the sustainable development goals , journal Journal of Cleaner Production volume 285 ( year 2021 ) pages 125319

    2021

  6. [6]

    Hajikhani , author A

    author A. Hajikhani , author A. Suominen , title Mapping the sustainable development goals (sdgs) in science, technology and innovation: application of machine learning in sdg-oriented artefact detection , journal Scientometrics volume 127 ( year 2022 ) pages 6661--6693

    2022

  7. [7]

    Maehara , author Y

    author Y. Maehara , author Y. Shiozawa , author Y. Osabe , title Multi-stage fine-tuning of patent domain-specific deberta for advanced patent landscape on sdgs/decarbonization , journal World Patent Information volume 81 ( year 2025 ) pages 102343

    2025

  8. [8]

    Ciarli , author A

    author T. Ciarli , author A. AlDoh , author S. Arora , author V. Arza , author J. Asinsten , author J. Assa , author J. Chataway , author A. Colonna , author H. Confraria , author P. N. Kombo , et al., title Changing directions: Steering science, technology and innovation towards the sustainable development goals ( year 2022 )

    2022

Show all 97 references
  1. [9]

    Callaert , author B

    author J. Callaert , author B. Van Looy , author A. Verbeek , author K. Debackere , author B. Thijs , title Traces of prior art: An analysis of non-patent references found in patent documents , journal Scientometrics volume 69 ( year 2006 ) pages 3--20

    2006

  2. [10]

    He , author M

    author Z.-L. He , author M. Deng , title The evidence of systematic noise in non-patent references: A study of new zealand companies’ patents , journal Scientometrics volume 72 ( year 2007 ) pages 149--166

    2007

  3. [11]

    Nagaoka , author I

    author S. Nagaoka , author I. Yamauchi , title The use of science for inventions and its identification: Patent level evidence matched with survey , journal Research Institute of Economy, Trade and Industry (RIETI) ( year 2015 )

    2015

  4. [12]

    Alstott , author G

    author J. Alstott , author G. Triulzi , author B. Yan , author J. Luo , title Mapping technology space by normalizing patent networks , journal Scientometrics volume 110 ( year 2017 ) pages 443--479

    2017

  5. [13]

    Zhai , author L

    author D. Zhai , author L. Zhai , author M. Li , author X. He , author S. Xu , author F. Wang , title Patent representation learning with a novel design of patent ontology: Case study on pem patents , journal Technological Forecasting and Social Change volume 183 ( year 2022 )...

    2022

  6. [14]

    Ghosh , author S

    author M. Ghosh , author S. Erhardt , author M. E. Rose , author E. Buunk , author D. Harhoff , title Paecter: Patent-level representation learning using citation-informed transformers , journal arXiv preprint arXiv:2402.19411 ( year 2024 )

    2024

  7. [15]

    Nicosia , author G

    author V. Nicosia , author G. Mangioni , author V. Carchiolo , author M. Malgeri , title Extending the definition of modularity to directed graphs with overlapping communities , journal Journal of Statistical Mechanics: Theory and Experiment volume 2009 ( year 2009 ) pages P03024

    2009

  8. [16]

    Li , author J

    author S. Li , author J. Hu , author Y. Cui , author J. Hu , title Deeppatent: patent classification with convolutional neural networks and word embedding , journal Scientometrics volume 117 ( year 2018 ) pages 721--744

    2018

  9. [17]

    Lee , author J

    author J.-S. Lee , author J. Hsiang , title Patentbert: Patent classification with fine-tuning a pre-trained bert model , journal arXiv preprint arXiv:1906.02124 ( year 2019 )

    1906 arXiv

  10. [18]

    Haghighian Roudsari , author J

    author A. Haghighian Roudsari , author J. Afshar , author W. Lee , author S. Lee , title Patentnet: multi-label classification of patent documents using deep learning based language understanding , journal Scientometrics volume 127 ( year 2022 ) pages 207--231

    2022

  11. [19]

    De Cunzo , author D

    author F. De Cunzo , author D. Consoli , author F. Perruchas , author A. Sbardella , title Mapping critical raw materials in green technologies , journal Industry and Innovation ( year 2025 ) pages 1--34

    2025

  12. [20]

    Giordano , author A

    author V. Giordano , author A. Castagnoli , author I. Pecorini , author F. Chiarello , title Identifying technologies in circular economy paradigm through text mining on scientific literature , journal PLoS One volume 19 ( year 2024 ) pages e0312709

    2024

  13. [21]

    Caldarola , author D

    author B. Caldarola , author D. Mazzilli , author L. Napolitano , author A. Patelli , author A. Sbardella , title Economic complexity and the sustainability transition: A review of data, methods, and literature , journal Journal of Physics: Complexity volume 5 ( year 2024 ) pa...

    2024

  14. [22]

    author G. S. Ascione , title Technological diversity to address complex challenges: the contribution of american universities to sdgs ( year 2023 )

    2023

  15. [23]

    Bordignon , title Dataset of search queries to map scientific publications to the un sustainable development goals , journal Data in Brief volume 34 ( year 2021 ) pages 106731

    author F. Bordignon , title Dataset of search queries to map scientific publications to the un sustainable development goals , journal Data in Brief volume 34 ( year 2021 ) pages 106731

    2021

  16. [24]

    Kashnitsky , author G

    author Y. Kashnitsky , author G. Roberge , author J. Mu , author K. Kang , author W. Wang , author M. Vanderfeesten , author M. Rivest , author L. Ke ler , author R. Jaworek , author M. Vignes , et al., title Identifying research supporting the united nations sustainable devel...

    2022 arXiv

  17. [25]

    Lampe , title Strategic citation , journal Review of Economics and Statistics volume 94 ( year 2012 ) pages 320--333

    author R. Lampe , title Strategic citation , journal Review of Economics and Statistics volume 94 ( year 2012 ) pages 320--333

    2012

  18. [26]

    Kuhn , author K

    author J. Kuhn , author K. Younge , author A. Marco , title Patent citations reexamined , journal The RAND Journal of Economics volume 51 ( year 2020 ) pages 109--132

    2020

  19. [27]

    Voskuil , author S

    author K. Voskuil , author S. Verberne , title Improving reference mining in patents with bert , journal arXiv preprint arXiv:2101.01039 ( year 2021 )

    2021 arXiv

  20. [28]

    Bekamiri , author D

    author H. Bekamiri , author D. S. Hain , author R. Jurowetzki , title Patentsberta: A deep nlp based hybrid model for patent distance and classification using augmented sbert , journal Technological Forecasting and Social Change volume 206 ( year 2024 ) pages 123536

    2024

  21. [29]

    Chen , author N

    author X. Chen , author N. Deng , title A semi-supervised machine learning method for chinese patent effect annotation , in: booktitle 2015 international conference on cyber-enabled distributed computing and knowledge discovery , organization IEEE , year 2015 , pp. pages 243--250

    2015

  22. [30]

    Huang , author J

    author L. Huang , author J. Yu , author Y. Hu , author H. Chang , title A semi-supervised learning framework for triz-based chinese patent classification , in: booktitle Proceedings of the 2020 6th International Conference on Computing and Artificial Intelligence , year 2020 ,...

    2020

  23. [31]

    Zhang , title Interactive patent classification based on multi-classifier fusion and active learning , journal Neurocomputing volume 127 ( year 2014 ) pages 200--205

    author X. Zhang , title Interactive patent classification based on multi-classifier fusion and active learning , journal Neurocomputing volume 127 ( year 2014 ) pages 200--205

    2014

  24. [32]

    Xiong , author S

    author S. Xiong , author S. Chen , author J. He , author Y. Liu , author J. Mao , author C. Liu , title Scalable multi-label patent classification via iterative large language model-assisted active learning , journal World Patent Information volume 82 ( year 2025 ) pages 102380

    2025

  25. [33]

    Edwards , author J

    author A. Edwards , author J. Camacho-Collados , title Language models for text classification: Is in-context learning enough? , journal arXiv preprint arXiv:2403.17661 ( year 2024 )

    2024 arXiv

  26. [34]

    Zhong , author S

    author M. Zhong , author S. Ouyang , author M. Jiang , author V. Hu , author Y. Jiao , author X. Wang , author J. Han , title Reactie: Enhancing chemical reaction extraction with weak supervision , journal arXiv preprint arXiv:2307.01448 ( year 2023 )

    2023 arXiv

  27. [35]

    Halpern , author S

    author Y. Halpern , author S. Horng , author D. Sontag , title Clinical tagging with joint probabilistic models , in: booktitle Machine Learning for Healthcare Conference , organization PMLR , year 2016 , pp. pages 209--225

    2016

  28. [36]

    author J. B. Reitsma , author A. W. Rutjes , author K. S. Khan , author A. Coomarasamy , author P. M. Bossuyt , title A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard , journal Journal of clinical epidemiology volume 62 ( ye...

    2009

  29. [37]

    Xu , author Y

    author S. Xu , author Y. Lu , author G. Schoenebeck , author Y. Kong , title Benchmarking llms' judgments with no gold standard , journal arXiv preprint arXiv:2411.07127 ( year 2024 )

    2024 arXiv

  30. [38]

    Bernhard-Harrer , author R

    author J. Bernhard-Harrer , author R. Ashour , author J.-M. Eberl , author P. Tolochko , author H. Boomgaarden , title Beyond standardization: a comprehensive review of topic modeling validation methods for computational social science research , journal Political Science Rese...

    2025

  31. [39]

    Ratner , author S

    author A. Ratner , author S. H. Bach , author H. Ehrenberg , author J. Fries , author S. Wu , author C. Ré , title Snorkel: rapid training data creation with weak supervision , journal Proceedings of the VLDB Endowment volume 11 ( year 2017 ) pages 269–282 . http://dx.doi.org/...

    2017

  32. [40]

    author X. J. Zhu , title Semi-supervised learning literature survey ( year 2005 )

    2005

  33. [41]

    Kim , author S

    author J. Kim , author S. Lee , title Technology opportunity analysis for creating innovative solutions: Applying semi-supervised topic modelling on patent data , in: booktitle 2024 Portland International Conference on Management of Engineering and Technology (PICMET) , organi...

    2024

  34. [42]

    Grootendorst , title Bertopic: Neural topic modeling with a class-based tf-idf procedure , journal arXiv preprint arXiv:2203.05794 ( year 2022 )

    author M. Grootendorst , title Bertopic: Neural topic modeling with a class-based tf-idf procedure , journal arXiv preprint arXiv:2203.05794 ( year 2022 )

    2022 arXiv

  35. [43]

    Settles , title Active learning literature survey ( year 2009 )

    author B. Settles , title Active learning literature survey ( year 2009 )

    2009

  36. [44]

    Narin , author K

    author F. Narin , author K. S. Hamilton , author D. Olivastro , title The increasing linkage between us technology and public science , journal Research policy volume 26 ( year 1997 ) pages 317--330

    1997

  37. [45]

    Marx , author A

    author M. Marx , author A. Fuegi , title Reliance on science: Worldwide front-page patent citations to scientific articles , journal Strategic Management Journal volume 41 ( year 2020 ) pages 1572--1594

    2020

  38. [46]

    Marx , author A

    author M. Marx , author A. Fuegi , title Reliance on science by inventors: Hybrid extraction of in-text patent-to-article citations , journal Journal of Economics & Management Strategy volume 31 ( year 2022 ) pages 369--392

    2022

  39. [47]

    Bedard-Vallee , author C

    author A. Bedard-Vallee , author C. James , author G. Roberge , title Elsevier 2023 sustainable development goals (sdgs) mapping , journal Elsevier Data Repository volume 1 ( year 2023 )

    2023

  40. [48]

    Kashnitsky , author G

    author Y. Kashnitsky , author G. Roberge , author J. Mu , author K. Kang , author W. Wang , author M. Vanderfeesten , author M. Rivest , author S. Chamezopoulos , author R. Jaworek , author M. Vignes , et al., title Evaluating approaches to identifying research supporting the ...

    2024

  41. [49]

    author K. A. Bryan , author Y. Ozcan , author B. Sampat , title In-text patent citations: A user's guide , journal Research Policy volume 49 ( year 2020 ) pages 103946

    2020

  42. [50]

    Ke , title An analysis of the evolution of science-technology linkage in biomedicine , journal Journal of Informetrics volume 14 ( year 2020 ) pages 101074

    author Q. Ke , title An analysis of the evolution of science-technology linkage in biomedicine , journal Journal of Informetrics volume 14 ( year 2020 ) pages 101074

    2020

  43. [51]

    Magerman , author B

    author T. Magerman , author B. Van Looy , author K. Debackere , title Does involvement in patenting jeopardize one’s academic footprint? an analysis of patent-paper pairs in biotechnology , journal Research Policy volume 44 ( year 2015 ) pages 1702--1713

    2015

  44. [52]

    Motohashi , author H

    author K. Motohashi , author H. Koshiba , author K. Ikeuchi , title Measuring science and innovation linkage using text mining of research papers and patent information , journal Scientometrics volume 129 ( year 2024 ) pages 2159--2179

    2024

  45. [53]

    Wang , author L

    author Y. Wang , author L. Pei , author J. Sun , author L. Kang , title Trace on both sides: a two-step text mining method to identify academic inventors’ patent--paper pairs , journal Scientometrics ( year 2024 ) pages 1--28

    2024

  46. [54]

    Cohen-Wang , author S

    author B. Cohen-Wang , author S. Mussmann , author A. Ratner , author C. R \'e , title Interactive programmatic labeling for weak supervision , in: booktitle Proceedings of the KDD DCCL Workshop, Anchorage, AK, USA , year 2019 , pp. pages 4--8

    2019

  47. [55]

    Denham , author E

    author B. Denham , author E. M. Lai , author R. Sinha , author M. A. Naeem , title Witan: unsupervised labelling function generation for assisted data programming , journal Proceedings of the VLDB Endowment volume 15 ( year 2022 ) pages 2334--2347

    2022

  48. [56]

    Block , author M

    author C. Block , author M. Wustmans , author N. Laibach , author S. Br \"o ring , title Semantic bridging of patents and scientific publications--the case of an emerging sustainability-oriented technology , journal Technological Forecasting and Social Change volume 167 ( year...

    2021

  49. [57]

    Shibata , author Y

    author N. Shibata , author Y. Kajikawa , author I. Sakata , title Detecting potential technological fronts by comparing scientific papers and patents , journal Foresight volume 13 ( year 2011 ) pages 51--60

    2011

  50. [58]

    Lippert , author K

    author K. Lippert , author K. U. F \"o rstner , title Patent-publication pairs for the detection of knowledge transfer from research to industry: reducing ambiguities with word embeddings and references , journal arXiv preprint arXiv:2412.00978 ( year 2024 )

    2024 arXiv

  51. [59]

    Masclans-Armengol , author S

    author R. Masclans-Armengol , author S. Hasan , author W. M. Cohen , title Measuring the Commercial Potential of Science , type Technical Report , Working Paper w32262, National Bureau of Economic Research, year 2024

    2024

  52. [60]

    Beltagy , author K

    author I. Beltagy , author K. Lo , author A. Cohan , title Scibert: A pretrained language model for scientific text , journal arXiv preprint arXiv:1903.10676 ( year 2019 )

    1903 arXiv

  53. [61]

    Cohan , author S

    author A. Cohan , author S. Feldman , author I. Beltagy , author D. Downey , author D. S. Weld , title Specter: Document-level representation learning using citation-informed transformers , journal arXiv preprint arXiv:2004.07180 ( year 2020 )

    2004 arXiv

  54. [62]

    Singh , author M

    author A. Singh , author M. D'Arcy , author A. Cohan , author D. Downey , author S. Feldman , title Scirepeval: A multi-format benchmark for scientific document representations , journal arXiv preprint arXiv:2211.13308 ( year 2022 )

    2022 arXiv

  55. [63]

    Guellec , author J

    author D. Guellec , author J. Liu , author M. Amdaoud , author W. Mescheba , author J. Quemener , author D. Sapinho , author J.-M. Deltorn , title Npl v. nlp: Analysing the links between science and technology with citations and semantics , journal NLP: Analysing the Links Bet...

    2024

  56. [64]

    Xu , author D

    author S. Xu , author D. Zhai , author F. Wang , author X. An , author H. Pang , author Y. Sun , title A novel method for topic linkages between scientific publications and patents , journal Journal of the Association for Information Science and Technology volume 70 ( year 201...

    2019

  57. [65]

    Dagdelen , author A

    author J. Dagdelen , author A. Dunn , author S. Lee , author N. Walker , author A. S. Rosen , author G. Ceder , author K. A. Persson , author A. Jain , title Structured information extraction from scientific text with large language models , journal Nature Communications volum...

    2024

  58. [66]

    author I. C. Wiest , author F. Wolf , author M.-E. Le mann , author M. van Treeck , author D. Ferber , author J. Zhu , author H. Boehme , author K. K. Bressem , author H. Ulrich , author M. P. Ebert , et al., title Llm-aix: An open source pipeline for information extraction fr...

    2024

  59. [67]

    Han , author C

    author R. Han , author C. Yang , author T. Peng , author P. Tiwari , author X. Wan , author L. Liu , author B. Wang , title An empirical study on information extraction using large language models , journal arXiv preprint arXiv:2305.14450 ( year 2023 )

    2023 arXiv

  60. [68]

    Yoshikawa , author R

    author N. Yoshikawa , author R. Krestel , title Do large language models understand patents? enhancing patent classification through ai-generated summaries , journal World Patent Information volume 81 ( year 2025 ) pages 102353

    2025

  61. [69]

    Hain , author R

    author D. Hain , author R. Jurowetzki , author M. Squicciarini , title Mapping complex technologies via science-technology linkages; the case of neuroscience--a transformer based keyword extraction approach , journal arXiv preprint arXiv:2205.10153 ( year 2022 )

    2022 arXiv

  62. [70]

    Achiam , author S

    author OpenAI , author J. Achiam , author S. Adler , author S. Agarwal , author L. A. et al. , title Gpt-4 technical report , year 2024 . https://arxiv.org/abs/2303.08774. http://arxiv.org/abs/2303.08774 arXiv:2303.08774

    2024 arXiv

  63. [71]

    Geng , title Label distribution learning , year 2016

    author X. Geng , title Label distribution learning , year 2016 . https://arxiv.org/abs/1408.6027. http://arxiv.org/abs/1408.6027 arXiv:1408.6027

    2016 arXiv

  64. [72]

    Akiba , author S

    author T. Akiba , author S. Sano , author T. Yanase , author T. Ohta , author M. Koyama , title Optuna: A next-generation hyperparameter optimization framework , year 2019 . https://arxiv.org/abs/1907.10902. http://arxiv.org/abs/1907.10902 arXiv:1907.10902

    2019 arXiv

  65. [73]

    author A. F. Agarap , title Deep learning using rectified linear units (relu) , journal arXiv preprint arXiv:1803.08375 ( year 2018 )

    2018 arXiv

  66. [74]

    author S. E. Robertson , author S. Walker , author S. Jones , author M. M. Hancock-Beaulieu , author M. Gatford , et al., title Okapi at trec-3 , journal Nist Special Publication Sp volume 109 ( year 1995 ) pages 109

    1995

  67. [75]

    Devlin , author M.-W

    author J. Devlin , author M.-W. Chang , author K. Lee , author K. Toutanova , title Bert: Pre-training of deep bidirectional transformers for language understanding , in: booktitle Proceedings of the 2019 conference of the North American chapter of the association for computat...

    2019

  68. [76]

    Srebrovic , author J

    author R. Srebrovic , author J. Yonamine , title Leveraging the bert algorithm for patents with tensorflow and bigquery , journal White paper ( year 2020 )

    2020

  69. [77]

    Reimers , author I

    author N. Reimers , author I. Gurevych , title Sentence-bert: Sentence embeddings using siamese bert-networks , journal arXiv preprint arXiv:1908.10084 ( year 2019 )

    1908 arXiv

  70. [78]

    Moradi , author K

    author M. Moradi , author K. Blagec , author F. Haberl , author M. Samwald , title Gpt-3 models are poor few-shot learners in the biomedical domain , journal arXiv preprint arXiv:2109.02555 ( year 2021 )

    2021 arXiv

  71. [79]

    Mu , author B

    author Y. Mu , author B. P. Wu , author W. Thorne , author A. Robinson , author N. Aletras , author C. Scarton , author K. Bontcheva , author X. Song , title Navigating prompt complexity for zero-shot classification: A study of large language models in computational social sci...

    2023 arXiv

  72. [80]

    Dhananasekaran , author R

    author S. Dhananasekaran , author R. Palanivel , author S. Pappu , title Adsorption of methylene blue, bromophenol blue, and coomassie brilliant blue by -chitin nanoparticles , journal Journal of advanced research volume 7 ( year 2016 ) pages 113--124

    2016

  73. [81]

    McPherson , author L

    author M. McPherson , author L. Smith-Lovin , author J. M. Cook , title Birds of a feather: Homophily in social networks , journal Annual review of sociology volume 27 ( year 2001 ) pages 415--444

    2001

  74. [82]

    author I. Hwang , title Evolution of the collaborative innovation network in the korean ict industry: a patent-based analysis , journal Technology Analysis & Strategic Management volume 35 ( year 2023 ) pages 221--236

    2023

  75. [83]

    Bergeaud , author Y

    author A. Bergeaud , author Y. Potiron , author J. Raimbault , title Classifying patents based on their semantic content , journal PloS one volume 12 ( year 2017 ) pages e0176310

    2017

  76. [84]

    Ciotti , author M

    author V. Ciotti , author M. Bonaventura , author V. Nicosia , author P. Panzarasa , author V. Latora , title Homophily and missing links in citation networks , journal EPJ Data Science volume 5 ( year 2016 ) pages 7

    2016

  77. [85]

    Nomaler , author B

    author \"O . Nomaler , author B. Verspagen , title Greentech homophily and path dependence in a large patent citation network , publisher Maastricht Economic and Social Research Institute on Innovation and … , year 2019

    2019

  78. [86]

    Arts , author B

    author S. Arts , author B. Cassiman , author J. C. Gomez , title Text matching to measure patent similarity , journal Strategic Management Journal volume 39 ( year 2018 ) pages 62--84

    2018

  79. [87]

    Yoo , author C

    author Y. Yoo , author C. Jeong , author S. Gim , author J. Lee , author Z. Schimke , author D. Seo , title A novel patent similarity measurement methodology: Semantic distance and technological distance , journal arXiv preprint arXiv:2303.16767 ( year 2023 )

    2023 arXiv

  80. [88]

    Whalen , author A

    author R. Whalen , author A. Lungeanu , author L. DeChurch , author N. Contractor , title Patent similarity data and innovation metrics , journal Journal of Empirical Legal Studies volume 17 ( year 2020 ) pages 615--639

    2020

  81. [89]

    Fritz , author G

    author C. Fritz , author G. De Nicola , author S. Kevork , author D. Harhoff , author G. Kauermann , title Modelling the large and dynamically growing bipartite network of german patents and inventors , journal Journal of the Royal Statistical Society Series A: Statistics in S...

    2023

  82. [90]

    Parchomovsky , author R

    author G. Parchomovsky , author R. P. Wagner , title Patent portfolios , journal U. Pa. L. Rev. volume 154 ( year 2005 ) pages 1

    2005

  83. [91]

    author A. B. Jaffe , author M. Trajtenberg , author R. Henderson , title Geographic localization of knowledge spillovers as evidenced by patent citations , journal the Quarterly journal of Economics volume 108 ( year 1993 ) pages 577--598

    1993

  84. [92]

    Criscuolo , author B

    author P. Criscuolo , author B. Verspagen , title Does it matter where patent citations come from? inventor vs. examiner citations in european patents , journal Research policy volume 37 ( year 2008 ) pages 1892--1908

    2008

  85. [93]

    author O. A. Jefferson , author A. Jaffe , author D. Ashton , author B. Warren , author D. Koellhofer , author U. Dulleck , author A. Ballagh , author J. Moe , author M. DiCuccio , author K. Ward , et al., title Mapping the global influence of published research on industry an...

    2018

  86. [94]

    Sclar , author Y

    author M. Sclar , author Y. Choi , author Y. Tsvetkov , author A. Suhr , title Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting , journal arXiv preprint arXiv:2310.11324 ( year 2023 )

    2023 arXiv

  87. [95]

    Errica , author G

    author F. Errica , author G. Siracusano , author D. Sanvito , author R. Bifulco , title What did i do wrong? quantifying llms' sensitivity and consistency to prompt engineering , journal arXiv preprint arXiv:2406.12334 ( year 2024 )

    2024 arXiv

  88. [96]

    Dey , author S

    author P. Dey , author S. Merugu , author S. Kaveri , title Uncertainty-aware fusion: An ensemble framework for mitigating hallucinations in large language models , in: booktitle Companion Proceedings of the ACM on Web Conference 2025 , year 2025 , pp. pages 947--951

    2025

  89. [97]

    Agrawal , author S

    author M. Agrawal , author S. Hegselmann , author H. Lang , author Y. Kim , author D. Sontag , title Large language models are few-shot clinical information extractors , journal arXiv preprint arXiv:2205.12689 ( year 2022 )

    2022 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.