REVIEW 3 major objections 5 minor 35 references
Scientific formulae keep their meaning hidden until a shared embedding space is learned.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Formula syntax and textual semantics show weak direct correspondence but strong latent correlation; contrastive learning bridges the gap and lifts retrieval from ~5% to ~58% recall@10.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection The diagnostic is worth a round of review, but the central mismatch claim rests on an unspecified GINE initialization and a shaky permutation test. the 3 major comments →
Syntax Meets Semantics: Understanding Scientific Formulae
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central finding is a diagnostic split: on every direct measure, formula syntax and semantics behave like unrelated spaces — mean cosine 0.085, cross-modal R@10 below 0.02, Procrustes error 0.83 — while on latent measures they are strongly linked, with top CCA correlations of 0.85–0.91. The authors interpret this as a representation mismatch: the same information exists in both modalities but is organized in different geometric subspaces. They then apply a contrastive alignment loss (InfoNCE with symmetric cross-modal terms plus per-modality consistency terms) to project both the syntactic operator-tree embedding and the semantic text embedding into one space. The aligned space supports b
What carries the argument
The load-bearing machinery is the pairing of an operator-tree graph representation with a contrastive alignment objective. Each formula is converted from MathML-like markup into an operator tree whose nodes carry canonical types (symbol, operator, function) and whose edges carry semantic roles such as base/exponent or numerator/denominator; a message-passing graph network encodes this structure, while a transformer encodes aggregated textual descriptions. Correspondence is diagnosed with three levels of analysis: instance-level cosine and retrieval, global representational similarity (RSA, CKA), and canonical correlation analysis (CCA), which finds linear projections maximizing cross-space c
Load-bearing premise
The raw-alignment measurements assume the graph-based syntactic encoder is a meaningful, learned representation at the moment of comparison; the paper never states whether that network was pretrained or randomly initialized, so the near-zero cosine and retrieval numbers could partly reflect an untrained encoder.
What would settle it
Re-run the raw-alignment metrics with a graph encoder that is pretrained on a formula-similarity task and then frozen; if cosine similarity and R@10 rise sharply, the 'weak observable correspondence' finding is an artifact of encoder initialization rather than an inherent property of formula syntax and semantics.
If this is right
- Formula retrieval systems that rely on frozen syntactic embeddings inherit the weak direct alignment; explicit cross-modal training is needed to make meaning searchable.
- Standard contrastive learning suffices — no bespoke architecture is required, so existing systems can add a projection head over current encoders.
- Bidirectional retrieval becomes practical, enabling both 'given a formula, find its explanation' and 'given a concept description, find the formula'.
- Graph-based encoders with role- and position-aware edges give the strongest alignment, suggesting hierarchical structure carries signal worth preserving.
- The near-chance raw retrieval numbers imply that lexical or shallow overlap methods are unlikely to recover formula semantics without learned alignment.
Where Pith is reading between the lines
- The large gap between near-zero raw similarity and high CCA suggests a simple linear probe trained on syntax embeddings might already recover much of the semantic signal; this is directly testable with the released dataset.
- The same diagnostic battery — direct similarity, global representational similarity, and CCA — could be applied to other independently trained paired encoders (e.g., code and comments, images and captions) to decide when alignment training is necessary.
- Because the gold dataset contains only 2,405 formula–concept pairs, the measured gains may not transfer unchanged to larger, noisier corpora; scaling behavior is an open question the paper does not address.
- The high latent correlation hints that a well-initialized graph encoder might already contain a readable semantic subspace, which would weaken the claim that syntax and semantics are 'extremely weakly' related in principle rather than in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the relationship between syntactic and semantic representations of scientific formulae. Syntactic representations are computed by converting content MathML into operator trees and encoding these with a GINE graph neural network; semantic representations are obtained by aggregating SentenceTransformer embeddings of Wikidata descriptions of the concepts linked to each formula. On a newly constructed dataset of 2,405 formula–concept pairs, the authors report very weak direct correspondence (mean cosine 0.085, R@10 < 0.02), moderate global similarity (RSA ρ=0.12, linear CKA≈0.16), but high canonical correlation (mean CCA 0.82, top-10 0.85–0.91). They then train contrastive projection heads over these encoders and report large improvements in cross-modal retrieval (Graph–Text CL R@10≈0.58 vs BM25≈0.05), concluding that explicit representation learning can recover correspondence that is not directly accessible in the native spaces.
Significance. If the empirical finding is valid, it provides a useful characterization for the math information retrieval community: formula syntax and semantics are not trivially aligned in off-the-shelf embedding spaces, but the shared signal is recoverable by standard contrastive learning. The paper has clear strengths: the dataset and code are publicly released, the retrieval experiments use 5-fold cross-validation with reported standard deviations, and comparisons include meaningful baselines (BM25, pretrained encoders without contrastive learning, and two dual-encoder variants). The conclusion that improvement is largely due to explicit alignment rather than a particular encoder is supported by consistency across architectures. However, the diagnostic claim that the native spaces exhibit 'extremely weak observable correspondence despite strong latent correlation' rests on experimental details that are currently under-specified and, in one case, internally inconsistent, so the central motivating evidence is not yet fully established.
major comments (3)
- [§3.1 and §4.2] The paper never states whether the GINE syntactic encoder used in the raw correspondence analysis was randomly initialized, pretrained, or trained in some other way. Section 4.2 is explicitly described as measuring 'native' representation spaces before any learned alignment, but if the GINE weights are random, the output is noise rather than a meaningful syntactic representation. The near-zero cosine/R@10 values and even the high CCA could then be artifacts of comparing a random projection to a pretrained text encoder. Please state the initialization and training status of the GINE at the point of measurement, and ideally include a control with a deliberately random GINE to show that the observed pattern differs from a noise baseline.
- [Table 1 and §4.2.1] The table reports a Permutation Test value of .0847 while the text says the permutation test shows cosine alignment is 'significantly above chance (0.08)'. If .0847 is a p-value, it is not significant at the 0.05 level, directly contradicting the claim that the weak correspondence is non-randomly above chance. If .0847 is instead the observed cosine or some other statistic, this must be clarified and the p-value reported. As currently written, the evidence for 'weak but non-random correspondence' is internally inconsistent.
- [§4.2, CCA analysis] The CCA results are reported as mean 0.816 with a top-10 range, but no uncertainty, permutation baseline, or cross-validation is provided. With 2,405 samples and high-dimensional embeddings (e.g., 768-dim), sample CCA can be substantially inflated even for independent random matrices. To support the load-bearing claim that the two modalities share 'strong latent structure', please report chance-level CCA (e.g., by permuting pair identities), confidence intervals, or a split-half analysis. Without such a baseline, the contrast between weak direct alignment and high CCA may reflect finite-sample bias rather than a genuine geometric mismatch.
minor comments (5)
- [§3.1] Typo: 'capturing semantic via its syntax' should be 'capturing semantics via its syntax'.
- [Figure 2] The right panel's axis label 'Formulas' and the use of 'Structural Semantic Similarity Gap' are unclear; please clarify what is plotted (e.g., pairwise distances or similarities, and whether values are per-formula or per-pair).
- [§4.2.1] The text says R@10 < 0.02, which is consistent with Table 1 (R@10 = .0125), but it would be useful to report standard deviations or confidence intervals for the raw retrieval figures, as the later retrieval experiments do.
- [§4.3] The definition of the intra-modal losses L_struct and L_sem is called InfoNCE, but the earlier cross-modal loss uses a symmetric CE/InfoNCE formulation. Please clarify whether the intra-modal losses use the same temperature and whether they are summed with equal weight.
- [References] Reference [9] (Hu et al.) appears to mix the title and a URL fragment; please verify the bibliographic entry for GINEConv.
Circularity Check
No significant circularity: raw correspondence measures are independent of the target claims, contrastive alignment is evaluated on held-out folds, and the only self-citation is non-load-bearing.
full rationale
The paper's contribution is empirical rather than deductive, and I could not exhibit any step where a claimed prediction reduces to its own input by construction. In §4.2, the raw correspondence analysis (cosine, Recall@k, Procrustes, RSA, CKA, CCA) is computed from independently constructed syntactic (GINE/OPT) and semantic (SentenceTransformer) representations; none of these metrics is fitted to the later retrieval claim, and the weak observable correspondence is not a renamed restatement of the inputs. In §4.3, the contrastive projection heads are trained with an InfoNCE objective on paired train folds and then evaluated on held-out test folds via 5-fold cross-validation, so the retrieval improvements are genuine out-of-sample measurements rather than in-sample optima used as predictions. The only self-citation is reference [27] (Schubotz et al., 2018), which supports the innocuous statement that formulae are captured with LaTeX or MathML; it is not load-bearing for any central claim. Two validity concerns are worth noting but are not circularity: the paper never states whether the GINE syntactic encoder was pretrained or randomly initialized before §4.2, which affects the interpretability of the 'native' representation; and the CCA latent correlations are reported in-sample without a cross-validated or permutation baseline, so their magnitude may be inflated. These are methodological/omission issues, not reductions of a derivation to its assumptions. Accordingly, the honest finding is no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (4)
- contrastive temperature τ =
not reported in §4.3
- projection heads W_struct, W_sem =
learned during contrastive training
- GINE encoder weights (node-type/edge-role/position embeddings, GINEConv layers) =
unknown; possibly random at raw-analysis time
- batch size B and training epochs =
not in text
axioms (5)
- domain assumption Wikidata P2534/P7235 formula-concept links define ground-truth semantic correspondence for formulas.
- domain assumption The semantic meaning of a formula is adequately represented by the sum of embeddings of its linked concept descriptions.
- domain assumption Operator-tree conversion from contentMathML preserves the relevant syntax-semantic relation.
- standard math CCA on high-dimensional representations yields meaningful latent correlations at these dimensions.
- ad hoc to paper The GINE encoder used in raw analysis is a valid 'native' representation.
Cite this review
Pith. "Pith review of Syntax Meets Semantics: Understanding Scientific Formulae." pith.science (2026). https://pith.science/paper/EGSFPO4D
@misc{pith2026260802457,
author = {Pith},
title = {Pith review of: Syntax Meets Semantics: Understanding Scientific Formulae},
year = {2026},
howpublished = {\url{https://pith.science/paper/EGSFPO4D}},
note = {Machine review of arXiv:2608.02457}
}
read the original abstract
Scientific formulae are a fundamental component of scholarly communication, yet their dual nature -- as structured syntax and carriers of semantics -- remains underexplored in scholarly information retrieval. Although prior studies show that jointly modeling syntactic and semantic modalities improves retrieval performance, the relationship between their underlying representations has not been systematically investigated. In this work, we empirically study cross-modal correspondence between formula syntax and semantics. We find that their native representation spaces exhibit extremely weak observable correspondence despite strong latent correlation, indicating a substantial representation mismatch between the two modalities. We further evaluate whether this mismatch can be reduced using standard representation learning and alignment techniques. We represent syntactic structure using graph-based encoders and semantic information using text-based encoders, then apply contrastive learning to induce a shared representation space. Results show that the learned alignment substantially improves cross-modal retrieval, suggesting that explicit representation learning can recover correspondence absent from the original representation spaces.
Figures
Reference graph
Works this paper leans on
-
[1]
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 2270–2282. doi:10.18653/v1/2020.acl-main.207
-
[2]
Yifan Dai, Liangyu Chen, and Zihan Zhang. 2020. An N-ary Tree-based Model for Similarity Evaluation on Mathematical Formulae. InProceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2578–2584
2020
-
[3]
Kenny Davila and Richard Zanibbi. 2017. Layout and Semantics: Combining Representations for Mathematical Formula Search. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 1165–1168
2017
-
[4]
Kenny Davila, Richard Zanibbi, Andrew Kane, and Frank Wm. Tompa. 2016. Tangent-3 at the NTCIR-12 MathIR Task. InProceedings of the 12th NTCIR Con- ference on Evaluation of Information Access Technologies (NTCIR). 338–345
2016
-
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. InNAACL
2019
-
[6]
Zhangyin Feng, Daya Guo, et al . 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InEMNLP
2020
-
[7]
Liangcai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. 2017. Preliminary Exploration of Formula Embedding for Mathematical Information Retrieval: Can Mathematical Formulae be Embedded like A Natural Language? arXiv preprint arXiv:1707.05154(2017)
Pith/arXiv arXiv 2017
-
[8]
Harold Hotelling. 1936. Relations between two sets of variates.Biometrika28, 3/4 (1936), 321–377
1936
-
[9]
Weihua Hu, Tianyan Xie, Yong Yan, Robin Sommer, Yu Jiang, Li Zhang, George Karypis, Jian Sun, and Jure Leskovec. 2020. Strategies for pre-training graph neural networks. InInternational Conference on Learning Representations (ICLR). https://openreview.net/forum?id=rJe7CCsEx Introduces the Graph Isomorphism Network with Edge features (GINEConv) for message passing
2020
-
[10]
Michael Kohlhase. 2006. OMDoc: An open markup format for mathematical documents. InInternational Conference on Artificial Intelligence and Symbolic Computation. Springer, 32–52
2006
-
[11]
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. InInternational Conference on Machine Learning (ICML). PMLR, 3519–3529
2019
-
[12]
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. 2008. Represen- tational similarity analysis–connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience2 (2008)
2008
-
[13]
Kriste Krstovski and David M. Blei. 2018. Equation Embeddings. arXiv:1803.09123 [stat.ML] https://arxiv.org/abs/1803.09123
Pith/arXiv arXiv 2018
-
[14]
Pavan Kumar, Arun Agarwal, and Chakravarthy Bhagvati
P. Pavan Kumar, Arun Agarwal, and Chakravarthy Bhagvati. 2012. A Structure Based Approach for Mathematical Expression Retrieval. InProceedings of the 6th Multi-Disciplinary International Workshop on Artificial Intelligence (MIW AI). 23–34
2012
-
[15]
Ruyin Li and Xiaoyu Chen. 2025. SSEmb: A Joint Structural and Semantic Embed- ding Framework for Mathematical Formula Retrieval. arXiv:2508.04162 [cs.IR] https://arxiv.org/abs/2508.04162
Pith/arXiv arXiv 2025
-
[16]
Zichao Li. 2025. Formula-Text Cross-Retrieval: A Benchmarking Study of Dense Embedding Methods for Mathematical Information Retrieval. InProceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025), Marco Valentino, Deborah Ferreira, Mokanarangan Thayaparan, Leonardo Ranaldi, and Andre Freitas (Eds.). Association for Computat...
-
[17]
Oard, and Richard Zanibbi
Behrooz Mansouri, Douglas W. Oard, and Richard Zanibbi. 2022. Contextualized Formula Search Using Math Abstract Meaning Representation. InProceedings of Y. Susanti et al. the 31st ACM International Conference on Information & Knowledge Management (CIKM). 4329–4333
2022
-
[18]
Oard, Jian Wu, C
Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C. Lee Giles, and Richard Zanibbi. 2019. Tangent-CFT: An Embedding Model for Mathematical Formulas. InProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 11–18
2019
-
[19]
Bruce Miller and Deyan Ginev. [n. d.]. LaTeXML: A LaTeX to XML/HTML/- MathML Converter. https://math.nist.gov/~BMiller/LaTeXML/ Accessed: 2025- Oct-22
2025
-
[20]
Miller and Abdou S
Bruce R. Miller and Abdou S. Youssef. 2003. Technical Aspects of the Digi- tal Library of Mathematical Functions.Annals of Mathematics and Artificial Intelligence38, 1–3 (2003), 121–136
2003
-
[21]
Jozef Mišutka and Leo Galamboš. 2008. Extending Full Text Search Engine for Mathematical Content.Towards Digital Mathematics Library(2008), 55–67
2008
-
[22]
Vít Novotný and Michal Štefánik. 2022. Combining Sparse and Dense Information Retrieval. InProceedings of the Working Notes of CLEF 2022, Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (Eds.). CEUR-WS, 104–118. http://ceur-ws.org/Vol-3180/paper-06.pdf
2022
-
[23]
Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. Math- BERT: A Pre-Trained Model for Mathematical Formula Understanding. arXiv:2105.00377 [cs.CL] https://arxiv.org/abs/2105.00377
Pith/arXiv arXiv 2021
-
[24]
Alec Radford, Jong Wook Kim, Chris Hallacy, et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. InICML
2021
-
[25]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing. Association for Computational Linguistics. https://arxiv.org/abs/1908.10084
Pith/arXiv arXiv 2019
-
[26]
P. H. Schönemann. 1966. A generalized solution of the orthogonal Procrustes problem.Psychometrika31 (1966), 1–10
1966
-
[27]
Moritz Schubotz, André Greiner-Petter, Philipp Scharpf, Norman Meuschke, Howard S. Cohl, and Bela Gipp. 2018. Improving the Representation and Con- version of Mathematical Formulae by Considering their Textual Context. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries(Fort Worth, Texas, USA)(JCDL ’18). Association for Computing ...
arXiv 2018
-
[28]
Heffernan, Xintao Wu, and Dongwon Lee
Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil T. Heffernan, Xintao Wu, and Dongwon Lee. 2021. MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education.CoRRabs/2106.07340 (2021). arXiv:2106.07340 https://arxiv.org/abs/2106.07340
Pith/arXiv arXiv 2021
-
[29]
Yujin Song and Xiaoyu Chen. 2021. Searching for Mathematical Formulas Based on Graph Representation Learning. InProceedings of the 14th Conference on Intelligent Computer Mathematics (CICM). 137–152
2021
-
[30]
Abhinav Thanda, Ankit Agarwal, Kushal Singla, Aditya Prakash, and Abhishek Gupta. 2016. A Document Retrieval System for Math Queries. InProceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies (NTCIR). 346–353
2016
-
[31]
Lan, and Richard G
Zichao Wang, Andrew S. Lan, and Richard G. Baraniuk. 2021. Mathematical Formula Representation via Tree Embeddings. InProceedings of the Third Inter- national Workshop on Intelligent Textbooks (iTextbooks@AIED 2021) (CEUR Work- shop Proceedings, Vol. 2895), Sergey Sosnovsky, Peter Brusilovsky, Richard Bara- niuk, and Andrew Lan (Eds.). CEUR-WS.org, 121–13...
2021
-
[32]
Zichao Wang, Mengxue Zhang, Richard G. Baraniuk, and Andrew S. Lan. 2021. Scientific Formula Retrieval via Tree Embeddings. In2021 IEEE International Conference on Big Data (Big Data). 1493–1503. doi:10.1109/BigData52589.2021. 9671942
arXiv 2021
-
[33]
Michihiro Yasunaga and John D Lafferty. 2019. Topiceq: A joint topic and mathematical equation model for scientific texts. InAAAI, Vol. 33. 7394–7401
2019
-
[34]
Ke Yuan, Liangcai Gao, Yuehan Wang, Xiaohan Yi, and Zhi Tang. 2016. A math- ematical information retrieval system based on RankBoost. In2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL). 259–260
2016
-
[35]
Wei Zhong and Richard Zanibbi. 2019. Structural Similarity Search for Formulas Using Leaf-Root Paths in Operator Subtrees. InProceedings of the 41st European Conference on Information Retrieval (ECIR). 116–129
2019
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.