Pith. sign in

REVIEW 2 major objections 5 minor 58 references

AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that mathematics can be represented as a directed graph of definitions, theorems, and problems that assembles and updates itself automatically from natural-language sources using large language models and vector-database…

desk verdict A useful system-building paper whose headline reachability claim is confounded because the query vectors contain the very reference lists used as ground truth; the rest is a reasonable proposal in need of reruns and baselines. read the letter →

arxiv 2505.13406 v1 pith:FQ27B2RQ submitted 2025-05-19 cs.AI

Rong Bian , Yu Geng , Zijian Yang , Bing Cheng This is my paper · ORCID
classification cs.AI
keywords mathematicalknowledgegraphvectordatabaseLLMaugmentationin-contextlearningfuzzysearchfusionSBERTembeddingsautomaticupdate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AutoMathKG proposes turning mathematics into a directed graph whose vertices are Definition, Theorem, and Problem entities and whose edges are reference relationships tagged with tactic roles such as premise, definition, and deduction. The authors argue that such a graph can be assembled automatically from ProofWiki, textbooks, arXiv preprints, and TheoremQA, with large language models augmenting entity content and extracting links through in-context learning. To make searching cheap, the graph is mirrored in a vector database built with two SBERT embedding strategies, and two update mechanisms keep the graph growing: an LLM-based 'Math LLM' completes missing proofs and solutions, while a vector search plus LLM judgement merges new entities into existing ones or adds them as fresh nodes. The value of the proposal, if it holds, is a wide-coverage mathematical resource that updates itself instead of being frozen at corpus-construction time.

What carries the argument

The load-bearing object is the entity embedding that folds graph neighbourhoods into vector space. Each entity is turned into five descriptive sentences — title, field, contents, in-references, and out-references — and MathVD1 concatenates them before SBERT embedding, while MathVD2 embeds each field separately and takes a weighted sum with weights (0.5, 0.3, 0.1, 0.05, 0.05). The in-refs and out-refs sentences are the structural carriers: the paper's ablation shows that removing them degrades similar-entity retrieval, while keeping them lets the top retrieval for 'probability measure is subadditive' be Boole's inequality, a direct consequence recorded in the target's references. Around this embedding, the system layers a directed graph with nine tactic labels and two automatic-update mechanisms, one filling missing proofs via Math LLM and one merging or adding entities by LLM judgement over vector-search candidates.

What would settle it

On a random sample of the test entities, recompute the 5-hop Hits@q for a MathVD variant whose vectors exclude the in-refs and out-refs fields and compare it against an oracle that simply matches the target's neighbour titles as text. If the title-matching oracle matches or beats MathVD while the no-refs variant collapses, the neighbour-embedding fields, not a learned structural representation, are producing the reachability result.

Watch

Extended reading notes

Core claim

The central claim is that a mathematical knowledge graph can be built and sustained automatically when mathematics is represented as a directed graph of Definition, Theorem, and Problem entities connected by reference edges, with each entity stored in a JSON schema recording segmented content, tactic-labelled references, and incoming and outgoing neighbours. For similar-entity search, the paper embeds five descriptive fields — title, field, contents, in-refs, and out-refs — into SBERT vectors, either as one concatenated long text (MathVD1) or as a weighted sum of separately embedded fields (MathVD2). The neighbour fields, the paper argues, are what let cosine similarity retrieve entities that are structurally related rather than merely textually similar. The paper reports that both vector databases outperform five knowledge-graph embedding baselines on k-hop reachability queries, that human-rated precision for retrieving similar entities is around 95%, and that Math LLM, built from a 7-billion-parameter base with task adapters and self-calibration, produces mostly correct or correct-reasoning answers across six question categories.

Load-bearing premise

The load-bearing premise is that embedding the list of an entity's reference neighbours measures structural reachability rather than simply letting a query match quoted neighbour titles verbatim; if exact title overlap is doing the work, the reported advantage over knowledge-graph embedding baselines would not demonstrate a general structural-search ability.

Editorial extensions

If this is right

  • A mathematics knowledge base can be assembled from heterogeneous sources without manual curation; duplicates are detected by vector search and merged by LLM judgement.
  • Fuzzy search over mathematical statements becomes practical: a natural-language query like 'expectation in probability and statistics' returns relevant definitions, theorems, and problems in a single ranked list.
  • An LLM grounded in the graph can complete missing proof steps and problem solutions, so the resource grows while it is being used.
  • Because edges carry tactic labels, downstream systems can distinguish whether a reference functions as a premise, a definition, or a conclusion rather than treating all references as identical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I read the design as implying that any corpus with explicit reference environments can be absorbed without retraining the vector model, since the pipeline is rule-based extraction followed by LLM labelling; the update mechanism is source-agnostic in principle.
  • Because the reachability metric is computed over the same reference edges that are embedded in the neighbour fields, a sharper structural test would evaluate on held-out edges that were not present during vector construction; the paper does not report such a split.
  • The weighted-sum scheme suggests a tunable knob: shifting weight from contents toward neighbour lists should trade semantic relevance for structural proximity, and applications could calibrate the weights per task.
  • The stored tactic labels open an unexplored path to explanation generation, where a system walks a retrieved chain and tells the reader why each step follows — by premise, by lemma, or by deduction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents AutoMathKG, a natural-language mathematical knowledge graph built from ProofWiki, textbooks, arXiv papers, and TheoremQA, with vertices representing definitions, theorems, and problems and directed edges representing reference relationships. The construction pipeline uses rule-based extraction followed by Llama-2-7b in-context learning for augmentation, and the graph is stored in a JSON schema with three levels of information. For retrieval, MathVD constructs SBERT-based vector databases via two embedding strategies: MathVD1 concatenates title, field, contents, in-refs, and out-refs text, while MathVD2 weight-summarizes per-field embeddings. Automatic updates are handled by Math LLM for knowledge completion and by a MathVD-plus-LLM fusion mechanism for merging or adding entities. The main empirical claims are that MathVD achieves superior reachability-query performance over five knowledge-graph embedding baselines and that Math LLM demonstrates strong reasoning capability.

Significance. If the central claims held, the paper would provide a useful, practically deployable system: a broad-coverage, automatically updatable mathematical knowledge graph with an associated vector database and a reasoning assistant. The strengths are the explicit end-to-end pipeline, the detailed JSON entity schema, the inclusion of diverse sources, the documentation of ICL prompt templates, an ablation of the reference fields, external-query examples, and concrete case studies of knowledge completion and fusion. However, the headline quantitative claim about reachability queries is, by construction of the embeddings, confounded with direct textual overlap of neighbor titles, and the reasoning-capability evaluation lacks baseline comparisons and reliability metrics; both issues are load-bearing for the abstract's claims.

major comments (2)
  1. [Section 5.1 (Eqs. 1-2), Section 5.2 (Eq. 3), Section 7.3.1 (Eq. 4), Section 8.2, Table 13] The reachability-query evaluation is circular with respect to the information embedded in the vectors. In both MathVD1 and MathVD2, the entity description includes S4(v) = 'in references: v[in refs]' and S5(v) = 'out references: v[out refs]', so the stored titles of neighboring entities are part of every vector. The Hits@q metric in Eq. (4) is then computed over k-hop reachability in the very same directed reference graph that fills these fields. For a query entity and a candidate entity that are k-hop reachable through an intermediate entity, the intermediate title can appear verbatim in both vectors, so high cosine similarity can result from literal name overlap in the embedded adjacency lists rather than from any learned structural representation. The ablation in Section 8.2 confirms that removing the reference fields degrades retrieval, and Table 13 shows that the top VD1-all hit for 'probability measure is subadditive' is 'Boole's inequality', an entity that is directly listed in the target's outgoing references. This means the claimed superiority over the five KG embedding baselines in Table 7 is not established as a test of structural search. I request a 'no-refs' evaluation of MathVD against the same baselines, or an evaluation on held-out edges, or an evaluation that uses the trained scoring functions of the baselines (rather than only cosine similarity of entity embeddings), to break the circularity.
  2. [Section 7.3.2, Table 8, Figure 10] The evaluation of Math LLM's reasoning capability is not sufficient to support the abstract's claim of 'robust mathematical reasoning capability'. The results consist of ratings assigned by the authors on a manually designed 1-5 scale, with no comparison baseline (e.g., the unmodified gemma-7b-it base model), no inter-annotator agreement measure, no significance test, and no calibration of the rating scale. The average ratings in Table 8 range from approximately 3.05 to 3.78, which the text interprets positively, but without a baseline or reliability evidence these numbers do not establish robustness. I ask for a direct comparison with the base model and with at least a standard instruction-tuned baseline, plus reporting of annotation protocol and agreement if the manual scoring is retained.
minor comments (5)
  1. [Table 1] There is a typo in the table caption: 'Courpus' should be 'Corpus'.
  2. [Section 7.3.1, Eq. (4)] The symbol q is used both for the number of retrieved neighbors and in the name 'Hits@q'; please define explicitly whether Hits@q is precision at q (r divided by the number of retrieved entities) and note that this differs from the 'hits at k' convention used in link prediction.
  3. [Section 7.3.3] The Kolmogorov-Smirnov test is mentioned but no test statistic or p-value is reported, so the claim that the two precision distributions show no significant difference cannot be verified.
  4. [Fig. 4 and Table B3] The attribute name is written as 'references tactics' in Fig. 4 and Table B3 but as 'references_tactics' in the JSON example in Fig. E1; please standardize the naming.
  5. [Section 7.3.1, Table 7] No training details or hyperparameters for the PyKEEN baselines are given (number of epochs, negative sampling, embedding dimension is 384 but other settings are absent), which makes it difficult to assess whether the comparison is fair.

Circularity Check

1 steps flagged · score 6.0 of 10

Reachability claim is confounded by construction: neighbor lists are embedded into the vectors and then used as the ground truth for the same reachability metric.

  1. self definitional [Section 5.1 Eq. (1); Section 7.3.1 Eq. (4); Section 8.2]
    "S4(v) = “in ref erences : v[in ref s]”, S5(v) = “out ref erences : v[out ref s]” ... Two entity nodes A and B are considered k-hop reachable if they can be connected by no more than k directed edges in the graph. For any entity vector e, we retrieved the top q most similar entities using cosine similarity and counted the number of k-hop reachable entities ... Hits@q = r/q."

    The query and candidate vectors are built by concatenating the literal in-refs and out-refs adjacency lists (Eq. 1, S4/S5), while the ground-truth 'k-hop reachable' set is computed over exactly those same directed edges (Eq. 4). For a two-hop path A→C→B, the query vector for A contains C's title in its out-refs text and the candidate vector for B contains C's title in its in-refs text, so high cosine similarity can arise from direct textual overlap of the embedded neighbor list rather than from any learned structural representation. The paper's own ablation (Section 8.2) confirms this: removing the in-refs/out-refs fields degrades retrieval, and the displayed VD1-all top hit is a direct consequence listed in the target's references.

full rationale

The central construction of AutoMathKG—extracting entities, edges, tactic labels, and using LLMs for augmentation—is not circular in itself, and much of the paper is a legitimate engineering contribution. The circularity is concentrated in the load-bearing empirical claim of Section 7.3.1, namely that MathVD gives 'superior reachability query results ... compared to five baselines.' By Eq. (1), each entity vector contains the entity's in-refs and out-refs titles; by Eq. (4), the evaluation counts k-hop reachable neighbors over the very graph edges that populate those fields. For two-hop reachability, the common intermediate entity's title appears verbatim in both the query vector's out-refs and the candidate vector's in-refs, so exact text overlap, not structural reasoning, can drive the reported Hits@q advantage. The ablation in Section 8.2 openly states that 'the reference information provided by incoming and outgoing entities plays a crucial role in the vector representation of mathematical knowledge,' which supports the interpretation that the reachability result is largely an artifact of embedding the adjacency list. The comparison with TransE, KG2E, HoLE, R-GCN, and BoxE is therefore not a fair test of whether the vectors have learned graph structure; those baselines are evaluated as entity-embedding cosine similarity rather than with their native scoring functions, while MathVD directly receives the adjacency text as input. The Math LLM reasoning evaluation and the fuzzy-search relevance examples are separate and are not affected by this circularity. Overall, the paper is partially circular in its headline reachability claim, but the rest of the system retains independent content, so the score is 6 rather than higher.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The claims depend on the representational choice of a directed graph of Definition, Theorem, and Problem nodes, on unvalidated LLM-generated edge and tactic information, and on the premise that cosine similarity over text embeddings captures structural reachability. The only numeric free parameter in the method is the hand-chosen MathVD2 weight vector; the knowledge fusion mechanism also uses n=5 retrieved candidates. No new mathematical or physical entities are introduced.

free parameters (2)
  • MathVD2 weight vector W = (0.5, 0.3, 0.1, 0.05, 0.05)
    Chosen by the authors' intuition that 'contents' and 'title' are most discriminative (Section 5.2). This weighting directly determines MathVD2 vectors and all MathVD2 results; it is not derived from data or theory, and no sensitivity analysis is provided.
  • Number of retrieved candidates n in knowledge fusion = 5
    Set in Section 7.1 for the fusion mechanism; no sensitivity analysis or justification is given for this value.
assumptions (6)
  • domain assumption Mathematics can be modeled as a directed graph of Definition, Theorem, and Problem entities with reference edges.
    The paper takes this representational assumption as given in Sections 1 and 3; it determines what is extracted and stored, and it is not independently validated.
  • domain assumption Rule-based matching over LaTeX environments reliably extracts definitions and theorems from ProofWiki, textbooks, and arXiv papers.
    Section 4.3 Step 1 relies on the NaturalProofs parsing approach, but the paper provides no evaluation of extraction precision or recall on its new sources.
  • ad hoc to paper Llama-2-7b with in-context learning produces accurate 'bodylist' segmentations, reference lists, and tactic labels for math entities.
    The augmented attributes in Section 4.3 Step 2 form the basis for all downstream structure and embeddings, yet the paper reports no accuracy metrics or human evaluation of these LLM outputs.
  • domain assumption Cosine similarity between SBERT embeddings of entity descriptions captures mathematical relevance and graph reachability.
    This is the core premise of MathVD in Section 5 and is tested in Section 7.3.1, but the testing protocol is circular because the neighbor text is embedded into the vectors.
  • ad hoc to paper The manually designed 1-5 rating scale for Math LLM is a valid measure of mathematical reasoning capability.
    Section 7.3.2 uses subjective ratings with no inter-annotator agreement, no baseline comparison, and no error bars, so the resulting capability claim is not independently grounded.
  • ad hoc to paper The constructed graph edges are ground truth for reachability queries.
    Section 7.3.1 evaluates reachability over the same graph produced by the pipeline under evaluation, so the ground truth is not independent of the method being tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database." pith.science (2026). https://pith.science/paper/FQ27B2RQ

@misc{pith2026250513406,
  author       = {Pith},
  title        = {Pith review of: AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQ27B2RQ}},
  note         = {Machine review of arXiv:2505.13406}
}
read the original abstract

A mathematical knowledge graph (KG) presents knowledge within the field of mathematics in a structured manner. Constructing a math KG using natural language is an essential but challenging task. There are two major limitations of existing works: first, they are constrained by corpus completeness, often discarding or manually supplementing incomplete knowledge; second, they typically fail to fully automate the integration of diverse knowledge sources. This paper proposes AutoMathKG, a high-quality, wide-coverage, and multi-dimensional math KG capable of automatic updates. AutoMathKG regards mathematics as a vast directed graph composed of Definition, Theorem, and Problem entities, with their reference relationships as edges. It integrates knowledge from ProofWiki, textbooks, arXiv papers, and TheoremQA, enhancing entities and relationships with large language models (LLMs) via in-context learning for data augmentation. To search for similar entities, MathVD, a vector database, is built through two designed embedding strategies using SBERT. To automatically update, two mechanisms are proposed. For knowledge completion mechanism, Math LLM is developed to interact with AutoMathKG, providing missing proofs or solutions. For knowledge fusion mechanism, MathVD is used to retrieve similar entities, and LLM is used to determine whether to merge with a candidate or add as a new entity. A wide range of experiments demonstrate the advanced performance and broad applicability of the AutoMathKG system, including superior reachability query results in MathVD compared to five baselines and robust mathematical reasoning capability in Math LLM.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 25 canonical work pages

  1. [1]

    Fensel, D., Simsek, U., Angele, K., Huaman, E., K¨ arle, E., Panasiuk, O., Toma, I., Umbrich, J., Wahler, A.: Knowledge graphs methodology, tools and selected use cases (2020)

    work page 2020

  2. [2]

    In: Journal of Physics: Conference Series, vol

    Zou, X.: A survey on application of knowledge graph. In: Journal of Physics: Conference Series, vol. 1487, p. 012016 (2020). IOP Publishing

    work page 2020

  3. [3]

    International Journal of Mathematical Education in Science and Technology 31(3), 393–414 (2000)

    Zwaneveld, B.: Structuring mathematical knowledge and skills by means of knowl- edge graphs. International Journal of Mathematical Education in Science and Technology 31(3), 393–414 (2000)

    work page 2000

  4. [4]

    arXiv preprint arXiv:2310.13021 (2023)

    Zhang, C.E., Collins, K.M., Weller, A., Tenenbaum, J.B.: Ai for mathematics: A cognitive science perspective. arXiv preprint arXiv:2310.13021 (2023)

    arXiv 2023

  5. [5]

    In: Intelligent Computer Mathematics: 13th International Confer- ence, CICM 2020, Bertinoro, Italy, July 26–31, 2020, Proceedings 13, pp

    Szegedy, C.: A promising path towards autoformalization and general artificial intelligence. In: Intelligent Computer Mathematics: 13th International Confer- ence, CICM 2020, Bertinoro, Italy, July 26–31, 2020, Proceedings 13, pp. 3–20 (2020). Springer

    work page 2020

  6. [6]

    In: Informal Reasoning and Education (1991)

    Schoenfeld, A.H.: On mathematics as sense-making: An informal attack on the unfortunate divorce of formal and informal mathematics. In: Informal Reasoning and Education (1991)

    work page 1991

  7. [7]

    Lulu Press, Morrisville, North Carolina (2019)

    Megill, N.D., Wheeler, D.A.: Metamath: A Computer Language for Pure Math- ematics. Lulu Press, Morrisville, North Carolina (2019)

    work page 2019

  8. [8]

    In: Automated Deduction–CADE 28: 28th International Conference on Auto- mated Deduction, Virtual Event, July 12–15, 2021, Proceedings 28, pp

    Moura, L.d., Ullrich, S.: The lean 4 theorem prover and programming language. In: Automated Deduction–CADE 28: 28th International Conference on Auto- mated Deduction, Virtual Event, July 12–15, 2021, Proceedings 28, pp. 625–635 (2021). Springer

    work page 2021

Show all 58 references
  1. [9]

    In: International Conference on Machine Learning, pp

    Wu, Y., Rabe, M.N., Li, W., Ba, J., Grosse, R.B., Szegedy, C.: Lime: Learn- ing inductive bias for primitives of mathematical reasoning. In: International Conference on Machine Learning, pp. 11251–11262 (2021). PMLR

    2021

  2. [10]

    Princeton University Press, Princeton, New Jersey (2010)

    Gowers, T., Barrow-Green, J., Leader, I.: The Princeton Companion to Mathe- matics. Princeton University Press, Princeton, New Jersey (2010)

    2010

  3. [11]

    : Mathgraph: A knowledge graph for automatically solving mathematical exercises

    Zhao, T., Huang, Y., Yang, S., Luo, Y., Feng, J., Wang, Y., Yuan, H., Pan, K., Li, K., Li, H., et al. : Mathgraph: A knowledge graph for automatically solving mathematical exercises. In: Database Systems for Advanced Applications: 24th International Conference, DASFAA 2019, Ch...

    2019

  4. [12]

    arXiv preprint 33 arXiv:2104.01112 (2021)

    Welleck, S., Liu, J., Bras, R.L., Hajishirzi, H., Choi, Y., Cho, K.: Natu- ralproofs: Mathematical theorem proving in natural language. arXiv preprint 33 arXiv:2104.01112 (2021)

    2021 arXiv

  5. [13]

    arXiv preprint arXiv:2205.03772 (2022)

    Wang, J.: Math-kg: Construction and applications of mathematical knowledge graph. arXiv preprint arXiv:2205.03772 (2022)

    2022 arXiv

  6. [14]

    arXiv preprint arXiv:2311.12649 (2023)

    Horowitz, L., Paiva, V.: Mathgloss: Building mathematical glossaries from text. arXiv preprint arXiv:2311.12649 (2023)

    2023 arXiv

  7. [15]

    arXiv preprint arXiv:2309.05653 (2023)

    Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., Chen, W.: Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653 (2023)

    2023 arXiv

  8. [16]

    : Language models are few-shot learners

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. : Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)

    2020

  9. [17]

    Proceedings of the 25th International Conference on World Wide Web (2016)

    Pezoa, F., Reutter, J.L., Suarez, F., Ugarte, M., Vrgoˇ c, D.: Foundations of json schema. Proceedings of the 25th International Conference on World Wide Web (2016)

    2016

  10. [18]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  11. [19]

    arXiv preprint arXiv:1908.10084 (2019)

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  12. [20]

    arXiv preprint arXiv:2103.03874 (2021)

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., Steinhardt, J.: Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021)

    2021 arXiv

  13. [21]

    arXiv preprint arXiv:2302.12433 (2023)

    Azerbayev, Z., Piotrowski, B., Schoelkopf, H., Ayers, E.W., Radev, D., Avi- gad, J.: Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. arXiv preprint arXiv:2302.12433 (2023)

    2023 arXiv

  14. [22]

    arXiv preprint arXiv:2310.06786 (2023)

    Paster, K., Santos, M.D., Azerbayev, Z., Ba, J.: Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786 (2023)

    2023 arXiv

  15. [23]

    arXiv preprint arXiv:2312.17120 (2023)

    Wang, Z., Xia, R., Liu, P.: Generative ai for math: Part i–mathpile: A billion- token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120 (2023)

    2023 arXiv

  16. [24]

    arXiv preprint arXiv:2204.05660 (2022)

    Mishra, S., Mitra, A., Varshney, N., Sachdeva, B., Clark, P., Baral, C., Kalyan, A.: Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. arXiv preprint arXiv:2204.05660 (2022)

    2022 arXiv

  17. [25]

    arXiv preprint 34 arXiv:2308.01825 (2023)

    Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., Zhou, C.: Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint 34 arXiv:2308.01825 (2023)

    2023 arXiv

  18. [26]

    ArXiv abs/2303.17760 (2023)

    Li, G., Hammoud, H.A.A.K., Itani, H., Khizbullin, D., Ghanem, B.: Camel: Com- municative agents for ”mind” exploration of large language model society. ArXiv abs/2303.17760 (2023)

    2023 arXiv

  19. [27]

    Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., Xia, T.: Theoremqa: A theorem-driven question answering dataset. (2023). The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  20. [28]

    Advances in neural information processing systems 35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)

    2022

  21. [29]

    arXiv preprint arXiv:2211.12588 (2022)

    Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 (2022)

    2022 arXiv

  22. [30]

    Advances in Neural Infor- mation Processing Systems 36 (2024)

    Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., Berner, J.: Mathematical capabilities of chatgpt. Advances in Neural Infor- mation Processing Systems 36 (2024)

    2024

  23. [31]

    IEEE transactions on neural networks and learning systems 33(2), 494–514 (2021)

    Ji, S., Pan, S., Cambria, E., Marttinen, P., Philip, S.Y.: A survey on knowl- edge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems 33(2), 494–514 (2021)

    2021

  24. [32]

    In: International Semantic Web Conference, pp

    Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: Dbpedia: A nucleus for a web of open data. In: International Semantic Web Conference, pp. 722–735 (2007). Springer

    2007

  25. [33]

    697–706 (2007)

    Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge, pp. 697–706 (2007). Proceedings of the 16th international conference on World Wide Web

    2007

  26. [34]

    Artificial intelligence 194, 28–61 (2013)

    Hoffart, J., Suchanek, F.M., Berberich, K., Weikum, G.: Yago2: A spatially and temporally enhanced knowledge base from wikipedia. Artificial intelligence 194, 28–61 (2013)

    2013

  27. [35]

    Google Blog

    Singhal, A.: Introducing the Knowledge Graph: things, not strings. Google Blog. 2020-11-13 (2012). https://www.blog.google/products/search/ introducing-knowledge-graph-things-not/

    2012

  28. [36]

    Advances in Neural Information Processing Systems 33, 9459–9474 (2020) 35

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.-t., Rockt¨ aschel, T.,et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459–9474 (2020) 35

    2020

  29. [37]

    Proceedings of the VLDB Endowment 9(11), 840–851 (2016)

    Zheng, W., Zou, L., Peng, W., Yan, X., Song, S., Zhao, D.: Semantic sparql sim- ilarity search over rdf knowledge graphs. Proceedings of the VLDB Endowment 9(11), 840–851 (2016)

    2016

  30. [38]

    arXiv preprint arXiv:1301.3781 (2013)

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

    2013 arXiv

  31. [39]

    1532–1543 (2014)

    Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word repre- sentation, pp. 1532–1543 (2014). Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)

    2014

  32. [40]

    arXiv preprint arXiv:1810.04805 (2018)

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  33. [41]

    Advances in neural information processing systems 26 (2013)

    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Trans- lating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)

    2013

  34. [42]

    623–632 (2015)

    He, S., Liu, K., Ji, G., Zhao, J.: Learning to represent knowledge graphs with gaus- sian embedding, pp. 623–632 (2015). Proceedings of the 24th ACM international on conference on information and knowledge management

    2015

  35. [43]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Nickel, M., Rosasco, L., Poggio, T.: Holographic embeddings of knowledge graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30 (2016)

    2016

  36. [44]

    In: The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15, pp

    Schlichtkrull, M., Kipf, T.N., Bloem, P., Van Den Berg, R., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. In: The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15, pp. 593–607...

    2018

  37. [45]

    Advances in Neural Information Processing Systems 33, 9649–9661 (2020)

    Abboud, R., Ceylan, I., Lukasiewicz, T., Salvatori, T.: Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems 33, 9649–9661 (2020)

    2020

  38. [46]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  39. [47]

    : Palm: Scaling lan- guage modeling with pathways

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al. : Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research 24(240), 1–113 (2023)

    2023

  40. [48]

    arXiv preprint arXiv:2211.09085 (2022) 36

    Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., Stojnic, R.: Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022) 36

    2022 arXiv

  41. [49]

    arXiv preprint arXiv:2403.08295 (2024)

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi` ere, M., Kale, M.S., Love, J., et al.: Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    2024 arXiv

  42. [50]

    arXiv preprint arXiv:2209.11895 (2022)

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al.: In-context learning and induction heads. arXiv preprint arXiv:2209.11895 (2022)

    2022 arXiv

  43. [51]

    arXiv preprint arXiv:2010.03648 (2020)

    Saunshi, N., Malladi, S., Arora, S.: A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648 (2020)

    2020 arXiv

  44. [52]

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 (2021)

    2021 arXiv

  45. [53]

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., Zettle- moyer, L.: Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 (2022)

    2022 arXiv

  46. [54]

    Lampinen, A.K., Dasgupta, I., Chan, S.C., Matthewson, K., Tessler, M.H., Creswell, A., McClelland, J.L., Wang, J.X., Hill, F.: Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 (2022)

    2022 arXiv

  47. [55]

    arXiv preprint arXiv:2106.09685 (2021)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  48. [56]

    arXiv preprint arXiv:2310.17918 (2023)

    Zhao, Y., Yan, L., Sun, W., Xing, G., Meng, C., Wang, S., Cheng, Z., Ren, Z., Yin, D.: Knowing what llms do not know: A simple yet effective self-detection method. arXiv preprint arXiv:2310.17918 (2023)

    2023 arXiv

  49. [57]

    arXiv preprint arXiv:2402.14830 (2024)

    Mitra, A., Khanpour, H., Rosset, C., Awadallah, A.: Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830 (2024)

    2024 arXiv

  50. [58]

    arXiv preprint arXiv:2110.14168 (2021) 37

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021) 37

    2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.