Unsupervised Construction of Knowledge Graphs From Text and Code

James Fairbanks; Kun Cao

REVIEW 5 major objections 6 minor 16 references

Unsupervised Construction of Knowledge Graphs From Text and Code

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Unsupervised clustering links code identifiers to science concepts.

desk verdict A promising unsupervised pipeline for connecting code identifiers to text concepts, but the evaluation is too thin to back the central claim. read the letter →

arxiv 1908.09354 v1 pith:GLQFBWI3 submitted 2019-08-25 cs.LG cs.IRstat.ML

Kun Cao , James Fairbanks This is my paper · ORCID

classification cs.LGcs.IRstat.ML

keywords knowledgegraphconstructionunsupervisedlearningwordembeddingsUMAPDBSCANclusteringsemanticcodeanalysisscientificsourceopenscience

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes an unsupervised pipeline that builds a knowledge graph from a scientific textbook and its companion source code, linking natural-language concepts to function and variable names. It claims that practical naming conventions make code identifiers specific enough that word embeddings can connect them to concepts in the text. The pipeline extracts subject-verb-object triples, clusters the subjects in a UMAP-reduced embedding space with DBSCAN, and attaches code entities to object nodes when similarity passes a threshold. If this works broadly, scientists could automatically organize and compare open-source scientific software by the concepts it implements, without paired training data or a handcrafted ontology.

What carries the argument

The load-bearing mechanism is a three-stage embedding-and-clustering operation: word embeddings of phrase tokens; UMAP nonlinear dimensionality reduction before density-based clustering (DBSCAN) of subject nodes, which groups lexically different but conceptually similar phrases; and raw-space DBSCAN over object nodes, preserving syntactic variants that code names can match. The final step compares each function or variable name embedding with object nodes and adds an edge when cosine similarity exceeds a fixed threshold. The contrast between the two clustering spaces is the key design choice: UMAP gives coarse conceptual clusters for subjects, while the object space stays fine-grained so that code-to-concept matching remains possible.

What would settle it

Run the pipeline on a corpus where all identifiers have been replaced by arbitrary tokens such as f1, tmp, and x7 while the text is unchanged. If concept-code edges survive, the method's success does not depend on naming conventions; if they disappear, the naming-convention assumption is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that a knowledge graph over scientific concepts and code can be built entirely unsupervised: entities are not predefined but emerge from clustering word embeddings of phrases in the text, and code entities are matched to those clusters by cosine similarity of their names. The authors demonstrate this on epidemiological modeling text with scientific code, producing a graph with 93 subject nodes and 115 object nodes, where variables such as infected_individuals and Beta attach to concepts like an exposed infectious class and a rate. A second textbook introduces mostly disjoint clusters, and the conductance between the two corpora drops as the matching threshold rises, indicating a transition from broad cross-disciplinary edges to precise within-discipline edges. The result is a procedure for making the procedural knowledge in code addressable through the conceptual vocabulary of the literature.

Load-bearing premise

The whole mapping rests on the assumption that code identifiers are named consistently enough with the concepts they implement that their embeddings resemble the words used in prose; the authors themselves note that storage and placeholder variables degrade accuracy.

Editorial extensions

If this is right

A knowledge graph built this way lets a scientist search a code base by the concepts a model implements, rather than by file names or documentation.
Adding a new textbook to the graph yields a quantitative comparison: conductance between the two corpora measures how much interdisciplinary connection the construction introduces.
Because no paired text-code examples are needed, the method can be applied to any open-science corpus whose code follows descriptive naming conventions.
Threshold selection can be guided either by precision-recall against small hand labels or by the conductance transition observed when a second corpus is added.
Downstream model-suggestion systems could use the conceptual nodes to detect semantically similar models and support model transformation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The conductance transition between thresholds could serve as a label-free threshold selector on new corpora, though the paper only observes it on one second corpus.
Because the method relies on an English word-embedding space, multilingual or heavily abbreviated code would likely need an embedding trained on identifiers themselves; the paper's English-only setup is a scope limit, not a proof of impossibility.
A natural stress test is to run the pipeline on code with systematically anonymized identifiers: if the concept-code edges collapse, the naming-convention assumption is confirmed as the active ingredient.
The same architecture could be extended to link equations or figures in a textbook to code, since current extraction relies only on textual subject-verb-object triples and code signatures.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. It assembles word2vec, UMAP, and DBSCAN to link variable and function names in Julia code to concepts extracted from a textbook, and it does so without paired training data. That assembly is new and potentially useful. The second thing is that the paper's central claim—that this mapping actually works—is not supported by the evidence it presents.

The idea is genuinely worth attending to. The authors exploit the fact that pedagogical code tends to use meaningful names, and they demonstrate a plausible path from raw markdown and Jupyter notebooks to a knowledge graph with subject, verb, object triples and code entities attached. They are also transparent about the main limitation: the Future Work section openly states that performance depends on naming quality and that English-language embeddings may not fit code syntax. That honesty counts for something.

The soft spots are real and they are concentrated in Section 4. The authors hand-label a set of object-variable edges, then use that same labeled set to pick the 0.7 similarity threshold and to report precision versus recall. That is circular: the threshold is tuned on the test set, so any performance estimate is optimistic. Worse, there are no actual precision or recall numbers in the text—only a figure. There is no baseline, so we cannot tell whether the UMAP-plus-DBSCAN combination is doing anything that a simpler cosine threshold would not. The conductance measurement with the second textbook is an interesting idea, but it is presented as a demonstration without validation. The stress-test note is on target.

If the authors supplied numbers, a held-out evaluation, and a baseline, this could be a solid contribution. Right now it reads like a promising workshop paper rather than a finished result. I would send it to peer review because the approach is novel enough to deserve referee time, but a referee should push hard on the evaluation. The paper is for people building semantic search tools over scientific code, and as a starting point it has value. I would not cite it yet for its empirical claims, but I would maybe bring it to a reading group to fuel discussion on unsupervised code-to-text alignment.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes an unsupervised pipeline that extracts <subject, verb, object> triples from expository text in online textbooks, builds a text knowledge graph, extracts variable and function names from associated Julia code, and links code entities to text concepts via word embeddings combined with UMAP and DBSCAN clustering. The central claim is that this process can construct a knowledge graph connecting scientific text concepts to source code entities without paired training examples or a handcrafted ontology. Evaluation is based on a hand-labeled set of expected variable/object edges, a precision-recall curve, and a conductance analysis when a second textbook corpus is added.

Significance. If the pipeline's mapping were validated, it would be a useful contribution to semantic software engineering and open-science knowledge graph construction: it is unsupervised, it uses publicly available corpora, and it addresses a real gap in connecting prose descriptions to code. The paper also makes a genuine effort at grounded evaluation, including a hand-labeled ground truth set and an exploratory conductance analysis. However, the evidence presented is not sufficient to establish the central claim: the only quantitative evaluation is a precision-recall figure with no numeric values, the threshold is tuned on the same labels used for evaluation, no baseline is reported, and the admitted dependence on naming quality is not quantified. The contribution is therefore plausible but unproven.

major comments (5)

[Section 4, Figure 4] The paper reports no numeric precision or recall values, no size of the hand-labeled set, and no inter-annotator agreement. The sentence 'We selected a threshold value of 0.7 to give a good balance between precision and recall' is therefore not supportable from the text, and the reader cannot determine whether the operating point is above chance. This is load-bearing because the threshold controls the central claim that variable and function names are reliably associated with object concepts.
[Section 4] The threshold selection and the evaluation use the same hand-labeled ground truth. The hand-labeled edges define tp, fp, and fn, and the threshold value 0.7 is chosen as 'higher than the intersection point' of the precision-recall curve computed on those same edges. Consequently, the reported precision-recall behavior is an estimate fit to the labels, not an independent prediction. A held-out test set, cross-validation, or a separate tuning set is required.
[Sections 3 and 4] No baseline or ablation is reported. The paper claims a 'novel process for joint clustering' combining word embeddings, UMAP, and DBSCAN, but it does not compare against alternatives such as a fixed cosine-similarity threshold on raw word embeddings, k-means, or agglomerative clustering. Without such a comparison, the contribution of UMAP and DBSCAN to mapping quality is not established.
[Section 4, Table 1] The conductance analysis is presented as a second threshold-selection method, but the criteria conflict. The text first selects 0.7 from precision-recall, then claims that 'the transition point between 0.30 and 0.35' shows a method for choosing a threshold. The conductance values decrease monotonically with threshold, and no statistical test, error bar, or null model supports calling 0.30/0.35 a transition point. The two procedures would recommend different thresholds.
[Section 5, Future Work] The paper concedes that 'the performance of our model relies on the quality of function and variable names' and that terse placeholder variables diminish accuracy. Because the evaluation is conducted on Epirecipes, which the authors themselves describe as having 'high-quality' pedagogical naming, the results do not support the broader claim about the open-source ecosystem. The limitation is acknowledged but never quantified, so the scope of the central claim is unclear.

minor comments (6)

[Section 2] The phrase 'spaCy's small natural language processing model' should specify the exact model (e.g., en_core_web_sm) and version for reproducibility.
[Section 3] The distinction between UMAP/DBSCAN producing 'semantically significant clusters' for subjects and DBSCAN-only producing 'syntactically similar' object nodes is not operationalized; define the similarity measure used for objects.
[Figure 2 caption] The caption mentions 'the big red vertex' but does not explain what cluster centers are or how colors map to node types; state this for accessibility.
[Section 4] The hand-labeled set is described only as 'crafted by a group of peers'; report the number of annotators, the instructions, and inter-annotator agreement so the evaluation is interpretable.
[Table 1] The row said to be 'indicated in bold' is not actually bold in the typeset table; correct the formatting and mention the specific conductance values used for the claimed transition.
[Abstract and Section 3] The term 'joint clustering text concepts' is used, but the procedure clusters subject nodes and object nodes separately; reword to avoid implying a joint optimization.

Circularity Check

1 steps flagged · score 2.0 of 10

Pipeline is largely self-contained; the main circularity is that the similarity threshold is selected and evaluated on the same hand-labeled set.

fitted input called prediction [Section 4, Results and Discussion (threshold selection and precision/recall evaluation)]
"In order to assess the threshold value for our variable assignment, we crafted a set of ground truth labels that were hand-labeled by a group of peers. ... Evaluation was conducted with respect to these labeled sets in terms of precision versus recall at various thresholds ... We selected a threshold value of 0.7 to give a good balance between precision and recall for our knowledge graph applications."

The same hand-labeled set is used both to choose the edge-similarity threshold and to compute the precision/recall behavior that justifies that choice. Selecting 0.7 because it balances precision and recall on these labels means the reported operating point is fitted to the evaluation yardstick rather than predicted independently. No held-out labels, cross-validation, or separate test set is described, so the precision/recall evidence for the variable-to-concept mapping is optimistically biased and does not independently demonstrate that the mapping generalizes. This is an evaluation-circularity concern, not a derivation that reduces the whole KG construction to the labels; the clustering pipeline itself is independent of the labeled set.

full rationale

The central construction is not circular by definition: text-to-code association is performed by word embeddings, UMAP, and DBSCAN clustering, with variables and functions connected to object nodes when embedding similarity exceeds a threshold. No target quantity in that pipeline is defined in terms of the final knowledge graph. The self-citation to SemanticModels.jl appears only as motivation and is not load-bearing evidence for the method. The main circular step is in Section 4: threshold selection and precision/recall evaluation use the same hand-labeled ground truth, so the reported precision-recall trade-off is fitted rather than predictive. The paper also omits numeric precision/recall values and baselines, which weakens the empirical claim but is a correctness/rigor issue rather than a definitional circularity. The Future Work concession that performance relies on the quality of function and variable names is an acknowledged limitation, not a circular step. Overall, the derivation preserves independent content, and the only identifiable circularity is localized to threshold evaluation, so the score is 2.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several unstated assumptions about naming conventions, NLP parsing quality, and embedding geometry. The main free parameters are the DBSCAN epsilon and the edge similarity threshold, which are chosen by hand or tuned on the evaluation set. No new physical or ontological entities are introduced.

free parameters (5)

DBSCAN epsilon for subject clusters in UMAP space = 0.30
Chosen after observing that direct DBSCAN on high-dimensional embeddings produced too-restrictive clusters (Section 3).
Similarity threshold for variable/function to object edges = 0.70
Selected by locating the threshold where recall equals precision on the hand-labeled set, then choosing a higher value (Section 4).
Subject component size cutoff = 5
Subject components with node size 5 or lower were removed to eliminate extraneous concepts (Section 3).
UMAP hyperparameters = not specified
UMAP is applied but n_neighbors, min_dist, and metric are not reported, making the reduction non-reproducible (Section 3).
DBSCAN parameters for object clustering = not specified
Objects are clustered without UMAP but the epsilon and min_samples are not given (Section 3).

assumptions (4)

domain assumption Function and variable names reflect the concepts they implement.
Stated in the abstract and Introduction; the whole mapping from code to concepts depends on this. The authors acknowledge in Future Work that placeholder names diminish accuracy.
domain assumption The spaCy small NLP model produces correct subject-verb-object triples from textbook sentences.
Used in preprocessing (Section 2); no evaluation of parsing accuracy is provided.
domain assumption Word embedding distances capture semantic similarity between text phrases and code identifiers.
The method assumes that 'sir_ode' maps near 'SIR model' in embedding space; no calibration is given.
domain assumption UMAP preserves semantic cluster structure while removing superficial lexical variation.
The central motivation for the UMAP step (Section 3); no quantitative check that UMAP preserves distances of interest.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unsupervised Construction of Knowledge Graphs From Text and Code." pith.science (2026). https://pith.science/paper/GLQFBWI3

@misc{pith2026190809354,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Construction of Knowledge Graphs From Text and Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLQFBWI3}},
  note         = {Machine review of arXiv:1908.09354}
}

read the original abstract

The scientific literature is a rich source of information for data mining with conceptual knowledge graphs; the open science movement has enriched this literature with complementary source code that implements scientific models. To exploit this new resource, we construct a knowledge graph using unsupervised learning methods to identify conceptual entities. We associate source code entities to these natural language concepts using word embedding and clustering techniques. Practical naming conventions for methods and functions tend to reflect the concept(s) they implement. We take advantage of this specificity by presenting a novel process for joint clustering text concepts that combines word-embeddings, nonlinear dimensionality reduction, and clustering techniques to assist in understanding, organizing, and comparing software in the open science ecosystem. With our pipeline, we aim to assist scientists in building on existing models in their discipline when making novel models for new phenomena. By combining source code and conceptual information, our knowledge graph enhances corpus-wide understanding of scientific literature.

Figures

Figures reproduced from arXiv: 1908.09354 by the authors.

**Figure 2.** A small portion of the resulting knowledge graph. This portion of the knowledge graph shows the relationships between concepts in SIR modeling. The red vertices are concept nodes with the big red vertex representing a cluster center for concepts related to "These Models", and the blue nodes are source code variable nodes. In the bottom left of the figure, you can see that the infected_individuals is related to the … view at source ↗

**Figure 1.** Modern online textbooks contain markdown files with expository text and Jupyter notebooks with code and figures. These input formats are designed for interactive instruction. Our work constructs scientific knowledge graphs for augmenting scientific reasoning from these data sources. 1a) Example of online Epirecipes Cookbook [6] textbook markdown file with 1b) corresponding source code file [PITH_FULL_IMAGE:figures… view at source ↗

**Figure 3.** a, phrases like “The Model”, “The Model of”, and “The model in a closed population” should share a semantic relationship but are not connected by the clustering algorithm. Tuning the DBSCAN epsilon parameter resulted in a reduction of the number of phrases that are clustered as noise, but did not resolve the problem of combining semantically similar phrases. To tackle this problem, we apply a UMAP transformation t… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Precision vs Recall trade-off in this context. Knowledge graph construction often prefers high recall, low precision thresholds because false positives can be filtered out in the downstream learning or reasoning steps. In this application, true positives are <variable…

**Figure 5.** Figure 5: A portion of the knowledge graph extracted from two online textbooks: Epirecipes Cookbook [6] and Statistics with Julia: Fundamentals for Data Science, Machine Learning and Artificial Intelligence [11]. Note that the subgraphs corresponding to each textbook (Epirecip…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

[1]

A survey of machine learning for big code and naturalness

Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) , 51(4):81, 2018

work page 2018
[2]

A density-based algorithm for discovering clusters in large spatial databases with noise

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pages 226–231, 1996

work page 1996
[3]

Towards ontology-based natural lan- guage processing

Dominique Estival, Chris Nowak, and Andrew Zschorn. Towards ontology-based natural lan- guage processing. In Proceeedings of the Work- shop on NLP and XML (NLPXML-2004): RD- F/RDFS and OWL in Language Technology , pages 59–66. Association for Computational Lin- guistics, 2004

work page 2004
[4]

Seman- ticmodels.jl, 2018

James Fairbanks and other contributors. Seman- ticmodels.jl, 2018

work page 2018
[5]

NSEEN: Neural Semantic Embedding for Entity Normalization

Shobeir Fakhraei and Jose Luis Ambite. Nseen: Neural semantic embedding for entity normal- ization. arXiv preprint arXiv:1811.07514, 2018

work page Pith review arXiv 2018
[6]

Epirecipes text book

Simon Frost, Allyson Walsh, and Jade Thomp- son. Epirecipes text book

work page
[7]

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

1997
[8]

spacy 2: Natural language understanding with bloom em- beddings

Matthew Honnibal and Ines Montani. spacy 2: Natural language understanding with bloom em- beddings. Convolutional Neural Networks and Incremental Parsing, 2017

work page 2017

Show all 16 references

[9]

Deep code comment generation

Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. InProceedings of the 26th Conference on Program Comprehen- sion, pages 200–210. ACM, 2018

2018
[10]

Mapping text to knowledge graph entities using multi-sense lstms

Dimitri Kartsaklis, Mohammad Taher Pilehvar, and Nigel Collier. Mapping text to knowledge graph entities using multi-sense lstms. arXiv preprint arXiv:1808.07724, 2018

2018 arXiv
[11]

Statistics with julia: Fundamentals for data science, ma- chine learning and artiﬁcial intelligence., May 2019

Hayden Klok and Yoni Nazarathy. Statistics with julia: Fundamentals for data science, ma- chine learning and artiﬁcial intelligence., May 2019. 7

2019
[12]

Umap: Uniform manifold approxi- mation and projection for dimension reduction

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approxi- mation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

2018 arXiv
[13]

Eﬃcient estimation of word rep- resentations in vector space

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey Dean. Eﬃcient estimation of word rep- resentations in vector space. arXiv preprint arXiv:1301.3781, 2013

2013 arXiv
[14]

An overview of embed- ding models of entities and relationships for knowledge base completion

Dat Quoc Nguyen. An overview of embed- ding models of entities and relationships for knowledge base completion. arXiv preprint arXiv:1703.08098, 2017

2017 arXiv
[15]

Silhouettes: a graphical aid to the interpretation and validation of cluster analysis

Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65, 1987

1987
[16]

Simple- owl: a generative lexicon ontology for nlp and the semantic web

Antonio Toral, Monica Monachini, et al. Simple- owl: a generative lexicon ontology for nlp and the semantic web. In Workshop on Coopera- tive Construction of Linguistic Knowledge Bases (AIIA 2007), 2007. 8

2007

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

A survey of machine learning for big code and naturalness

Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) , 51(4):81, 2018

work page 2018

[2] [2]

A density-based algorithm for discovering clusters in large spatial databases with noise

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pages 226–231, 1996

work page 1996

[3] [3]

Towards ontology-based natural lan- guage processing

Dominique Estival, Chris Nowak, and Andrew Zschorn. Towards ontology-based natural lan- guage processing. In Proceeedings of the Work- shop on NLP and XML (NLPXML-2004): RD- F/RDFS and OWL in Language Technology , pages 59–66. Association for Computational Lin- guistics, 2004

work page 2004

[4] [4]

Seman- ticmodels.jl, 2018

James Fairbanks and other contributors. Seman- ticmodels.jl, 2018

work page 2018

[5] [5]

NSEEN: Neural Semantic Embedding for Entity Normalization

Shobeir Fakhraei and Jose Luis Ambite. Nseen: Neural semantic embedding for entity normal- ization. arXiv preprint arXiv:1811.07514, 2018

work page Pith review arXiv 2018

[6] [6]

Epirecipes text book

Simon Frost, Allyson Walsh, and Jade Thomp- son. Epirecipes text book

work page

[7] [7]

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

1997

[8] [8]

spacy 2: Natural language understanding with bloom em- beddings

Matthew Honnibal and Ines Montani. spacy 2: Natural language understanding with bloom em- beddings. Convolutional Neural Networks and Incremental Parsing, 2017

work page 2017