REVIEW 3 major objections 2 minor 23 references
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · grok-4.3
Co-occurrence networks from full-text papers show classic algorithms at research-period intersections hold high centrality and balanced influence.
2026-06-26 00:34 UTC pith:A5ZPBZ43
load-bearing objection They build co-occurrence networks from algorithm names in NLP papers over four decades but give no validation that the extraction or links actually track influence instead of topic overlap. the 3 major comments →
Exploring Academic Influence of Algorithms by Co-occurrence Network Based on Full-text of Academic Papers
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By constructing overall, cumulative, and annual co-occurrence networks of algorithms from full-text academic papers in natural language processing, the study establishes that these networks display typical features of complex networks with increasingly dense connections over time, and that classic, high-performing algorithms located at the intersections of different research periods exhibit high popularity, control, centrality, and balanced influence, while declining influence is marked first by loss of core network position followed by weaker associations.
What carries the argument
Algorithm co-occurrence networks extracted via deep learning entity recognition from full-text papers, then analyzed with multiple centrality measures across overall, cumulative, and yearly snapshots.
Load-bearing premise
That the co-occurrence of algorithm names within papers reliably signals meaningful influence connections instead of shared topics, citation practices, or extraction errors.
What would settle it
A longitudinal check showing that an algorithm's mention frequency and independent usage metrics (such as benchmark adoption) continue to rise after it has already lost its top centrality rank in the co-occurrence network would falsify the claim that core-position loss precedes influence decline.
If this is right
- Classic high-performing algorithms maintain elevated centrality scores across the entire field and across successive time windows.
- Algorithms positioned at the boundaries between research periods display more balanced influence than those confined to single eras.
- Loss of influence for any algorithm first appears as a drop in core network position before its links to other algorithms weaken.
- The networks become progressively denser, reflecting growing interconnections among algorithms over two decades.
Where Pith is reading between the lines
- The same extraction and centrality pipeline could be applied to track influence shifts in other scientific domains beyond NLP.
- Early detection of centrality loss might serve as a leading indicator for when an algorithm is about to lose practical adoption.
- Combining these networks with author or task co-occurrence graphs could reveal how scholars and problems drive algorithmic change.
- The approach supplies a structural baseline against which future claims of algorithmic impact can be compared without relying solely on citation counts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper extracts algorithm entities from the full text of NLP papers spanning more than four decades using deep learning models, constructs overall, cumulative, and annual co-occurrence networks, and analyzes their structural properties (e.g., density, complex-network features) together with multiple centrality measures to quantify collective algorithm influence. It reports that classic high-performing algorithms and those at research-period intersections exhibit high popularity, control, and balanced centrality, while declining influence is typically preceded by loss of core network position.
Significance. If the extraction and co-occurrence steps can be shown to reliably encode influence rather than topic overlap or noise, the work supplies the first large-scale temporal network view of algorithm influence in NLP, covering four decades and linking structural position to popularity and decline patterns. The scale and the distinction among overall/cumulative/annual networks are genuine strengths.
major comments (3)
- [Abstract / Methods] Abstract and Methods: the construction of all reported networks rests on deep-learning entity extraction, yet no precision, recall, F1, validation-set performance, inter-annotator agreement, or error analysis is supplied; without these figures the centrality rankings and temporal claims cannot be evaluated.
- [Results] Results (centrality and decline analysis): the headline claim that 'when the influence of an algorithm declines, it usually loses its core network position first' is presented as an observational finding, but the manuscript provides no independent measure of influence (e.g., citation counts, task adoption) against which network position can be validated, leaving open the possibility that co-occurrence simply tracks shared sub-topics.
- [Results / Discussion] Results / Discussion: no sensitivity analysis or controls are described for co-occurrence threshold choice, NER false-positive rate, or alternative explanations such as citation conventions; these choices directly affect the reported 'increasingly dense connections' and the identification of 'core positions.'
minor comments (2)
- [Abstract] The abstract states results on 'balanced influence' without defining the term or the exact centrality combination used; a short operational definition would improve clarity.
- [Figures] Figure captions and network-visualization panels should explicitly state the time windows and edge-weighting rule employed for each cumulative/annual snapshot.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments. We address each major point below and indicate the revisions we will make to strengthen the manuscript.
read point-by-point responses
-
Referee: [Abstract / Methods] Abstract and Methods: the construction of all reported networks rests on deep-learning entity extraction, yet no precision, recall, F1, validation-set performance, inter-annotator agreement, or error analysis is supplied; without these figures the centrality rankings and temporal claims cannot be evaluated.
Authors: We agree that the absence of quantitative validation for the entity extraction model is a limitation. In the revised manuscript we will add a Methods subsection reporting precision, recall, and F1 on a held-out validation set, together with an error analysis of false positives and negatives. This will allow readers to assess the reliability of the extracted entities before interpreting the network results. revision: yes
-
Referee: [Results] Results (centrality and decline analysis): the headline claim that 'when the influence of an algorithm declines, it usually loses its core network position first' is presented as an observational finding, but the manuscript provides no independent measure of influence (e.g., citation counts, task adoption) against which network position can be validated, leaving open the possibility that co-occurrence simply tracks shared sub-topics.
Authors: The reported patterns are observational and derived from the temporal evolution of network position. We accept that external validation would increase confidence. In revision we will add a supplementary analysis correlating network centrality with citation counts for a representative sample of algorithms. At the same time, the multi-network design (overall, cumulative, and annual) and the observed complex-network properties provide evidence that the co-occurrence relations capture more than simple topical overlap. revision: partial
-
Referee: [Results / Discussion] Results / Discussion: no sensitivity analysis or controls are described for co-occurrence threshold choice, NER false-positive rate, or alternative explanations such as citation conventions; these choices directly affect the reported 'increasingly dense connections' and the identification of 'core positions.'
Authors: We acknowledge that sensitivity checks and discussion of alternative explanations are missing. The revised version will include (i) sensitivity analyses varying the co-occurrence threshold and (ii) an assessment of how NER false-positive rates could affect density and core-position findings. We will also add a brief discussion of citation conventions as a potential confounder and explain how the annual-network construction helps isolate genuine temporal shifts. revision: yes
Circularity Check
No circularity: purely observational network statistics from extracted co-occurrences
full rationale
The paper extracts algorithm mentions via deep-learning NER, constructs co-occurrence networks, and reports standard network metrics (density, centrality) plus temporal trends. No equations, no fitted parameters renamed as predictions, no self-citation chains invoked as uniqueness theorems, and no ansatz or renaming of known results. All reported findings are direct descriptive statistics on the constructed graphs; the central claims about influence follow from the observed network positions without reduction to inputs by construction.
Axiom & Free-Parameter Ledger
read the original abstract
Algorithms have become central to scientific research in the era of artificial intelligence (AI). Although algorithm mentions in papers are often used to indicate popularity and influence, existing studies usually evaluate individual algorithms in isolation and pay limited attention to the collective influence formed through their interconnections. This study constructs large-scale algorithm co-occurrence networks in natural language processing (NLP) based on the full text of academic papers and investigates algorithm influence from a network perspective. Using deep learning models, we extract algorithm entities and build overall, cumulative, and annual co-occurrence networks. We analyze their structural characteristics and apply multiple centrality measures to assess the group influence of algorithms across the whole field and over time. The results show that algorithm networks display typical features of complex networks, with increasingly dense connections developing over approximately two decades. Classic, high-performing algorithms and those located at the intersections of different research periods tend to have high popularity, control, centrality, and balanced influence. When the influence of an algorithm declines, it usually loses its core network position first, followed by weaker associations with other algorithms. This study is the first large-scale analysis of algorithm co-occurrence networks. Covering more than four decades of academic publications, it provides a temporal and structural view of algorithm influence and offers a foundation for future research on networks linking algorithms, scholars, and tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Ding et al
ded the research object to the algorithm entities in the NLP domain from 1979 to 2015, and evaluated the academic influence of the algorithm via the number of articles and the duration of the mention. Ding et al. (2019) further analyzed the citation sentences of algorithm entities, and discussed the evolution of algorithm citation sentences over time, whe...
1979
-
[2]
Depsy:valuing the software that powers science
and analyzed the influence diffusion trends of software entities at the levels of literature, journals, and fields (Pan et al., 2018). Furthermore, Ma and Zhang (2017) used total mentions and average mentions to assess the academic influence of software entities, identifying typical roles of software in PLOS One. In addition to various mention metrics, al...
2018
-
[3]
Network attributes also bring new ideas to traditional frequency-based entity evaluation
is close, and whether there is knowledge linkage and exchange between different disciplines or fields (Song et al., 2023). Network attributes also bring new ideas to traditional frequency-based entity evaluation. Ding et al. (2009) ranked the importance of scholars in a co-citation network based on the PageRank algorithm. Fiala (2012) defined the co-autho...
2023
-
[4]
Some scholars considered the dynamic changes of networks that determine the network’s final structure and topological properties (Abbasi et al., 2012; Dorogovtsev & Mendes, 2002)
and optimize knowledge retrieval and recommendation. Some scholars considered the dynamic changes of networks that determine the network’s final structure and topological properties (Abbasi et al., 2012; Dorogovtsev & Mendes, 2002). Liu and Xia (2015) analyzed the development of cooperation between interdisciplinary fields. Liu and Guan (2015) further exp...
2012
-
[5]
We then proceeded to conduct evaluations of the impact of these algorithmic nodes within these networks
Methodology We demonstrated the process of extracting algorithm entities from academic papers and leveraging their co-occurrences to create diverse networks. We then proceeded to conduct evaluations of the impact of these algorithmic nodes within these networks. 3.1 Data collection The conference proceedings of the Annual Meeting of the Association for Co...
2020
-
[6]
Flow chart of the process of identifying algorithm entities in papers Training corpus (1979-2015) Dictionary of algorithms ((1979-2020) Candidate entities Algorithm list in each article Best model Model 1 Full-text content of articles (2016-2020) Model 2 Model n Uncertain entities Algorithm entities (2016-2020) Algorithm entities (1979-2015) Full-text con...
1979
-
[7]
algorithm,
Performance of the three extraction models Source(s): Table by authors Model Precision Recall F1 value BiLSTM+CRF 0.872 0.901 0.8862 BERT+CRF 0.953 0.962 0.9574 BERT+BiLSTM+CRF 0.970 0.959 0.9642 3.2.2 Identification of the algorithm entities We manually labeled the candidate entities to identify the exact algorithm entities. First, we removed 16,108 cand...
1979
-
[8]
CC finds the shortest path between each node first, then assigns a score to each node according to the sum of path length
was used to reflect the centrality or central position of algorithm nodes within a group. CC finds the shortest path between each node first, then assigns a score to each node according to the sum of path length. The CC is a proxy for the independence and efficiency of communicating with other nodes; therefore, it helps identify strategically positioned n...
2012
-
[9]
They exhibit notably short distances between nodes, significant clustering coefficients, and a degree distribution that resembles a power-law
Degree distribution of the co-occurrence network of algorithms Based on the findings above, we observe that algorithm co-occurrence networks share similarities with other complex networks, such as scholarly collaboration networks and actor networks (Albert and Barabási, 2000). They exhibit notably short distances between nodes, significant clustering coef...
2000
-
[10]
small world
The average degree of cumulative algorithm co-occurrence network The average length of the shortest path between each node pair is called the average separation. In Figure 4, the average separation experienced a substantial increase, peaking in 1994, followed by a steady decrease. This pattern suggests that the paths between algorithm pairs became longer ...
1994
-
[11]
Figure 5 illustrates the changes in the average clustering coefficient of the cumulative network, and data from 1979 and 1980 are excluded since the value is zero
Average separation distribution of the cumulative algorithm co-occurrence network by year The clustering coefficient reflects the degree of connection between all neighbor nodes of a focal node. Figure 5 illustrates the changes in the average clustering coefficient of the cumulative network, and data from 1979 and 1980 are excluded since the value is zero...
1979
-
[12]
The relative size increased markedly in the first five years, and after a brief decline from 1983 to 1986, it rose sharply again until
The clustering coefficient of the cumulative algorithm co-occurrence network by year Figure 6 presents the relative size of the giant component in each cumulative network. The relative size increased markedly in the first five years, and after a brief decline from 1983 to 1986, it rose sharply again until
1983
-
[13]
It can be speculated that the topology of the algorithm network stabilized after 2006, and almost all algorithms appeared in the giant component
From 1993-2006, the relative size gradually exceeded 95%. It can be speculated that the topology of the algorithm network stabilized after 2006, and almost all algorithms appeared in the giant component. Source(s): Figure by authors Figure
1993
-
[14]
gating mechanism
The relative size of the giant component in the cumulative algorithm co-occurrence network by year The above results show that the close correlation between algorithms in the field of NLP did not happen overnight. In the first 20 years, the structure of co-occurring networks formed between algorithms was not tight. After almost 20 years of turbulence, the...
2020
-
[15]
Top 10 algorithms in data mining
The top-5 algorithms with high influence based on each network indicators Source(s): Table by authors Type Rank Popularity Control Central position Balance Classification algorithm 1 Support vector machine Support vector machine Support vector machine Support vector machine 2 Maximum entropy Maximum entropy Maximum entropy Maximum entropy 3 Gibbs sampling...
2008
-
[16]
Algorithms with the highest influence each year based on network indicators Source(s): Table by authors Year Popularity Control Central position Balance 1979 ATN ATN ATN discrimination network 1980 ATN ATN ATN greedy regression 1981 ATN ATN ATN left corner 1982 ATN ATN ATN ATN 1983 CFG CFG CFG CFG 1984 LFG ATN LFG DCG 1985 CFG ATN CFG CFG 1986 CFG GPSG GP...
1979
-
[17]
Since only five years 20 had passed between the SVM being used and its appearance in ACL, it was not yet connected with other algorithms to achieve a higher group status
Unlike CFG, the influence of SVM in the network did not reach its peak as soon as it appeared. Since only five years 20 had passed between the SVM being used and its appearance in ACL, it was not yet connected with other algorithms to achieve a higher group status. Since 2002, the influence ranking of SVM began to increase significantly. In 2004, SVM rank...
2002
-
[18]
LSTM was initially proposed in 1997 (Hochreiter and Schmidhuber,1997)
The influence evolution of support vector machine (3) The influence evolution of long short-term memory 21 We selected long short-term memory (LSTM) as the representative deep learning algorithm. LSTM was initially proposed in 1997 (Hochreiter and Schmidhuber,1997). In the first ten years of the 21st century, scholars proposed some epoch-making models, su...
1997
-
[19]
At the end of 2018, the pre-training model BERT was proposed and set off a new round of deep learning model revolution in the NLP field
The decline in the ranking may be due to the increase of new algorithms, but the decrease in the score shows that other algorithms have shaken the position of LSTM. At the end of 2018, the pre-training model BERT was proposed and set off a new round of deep learning model revolution in the NLP field. Various algorithms derived from BERT are also emerging ...
2018
-
[20]
bridge builder,
Discussion (1) Influence ranking of algorithms based on traditional frequency indicator Following the existing method (Wang & Zhang, 2020), we calculated the academic influence of algorithms based on the number of papers mentioning them. Table 5 presents the top 10 algorithms ranked according to their frequency-based influence, where the most influential ...
2020
-
[21]
(3) The evolution rules of network influence We compared the influence ranking and evolution of different algorithms based on the network
Top three algorithm pairs in the algorithm co-occurrence network by year Source(s): Table by authors Rank Year 1 2 3 1979 ATN-Discrimination Network CFG-Local Maxima 1980 CFG-PSG ATN-Semantic Grammar ATN-Author Topic 1981 Left corner parsing-Semantic grammar ATN- Left corner parsing CYK- Left corner parsing 1982 ATN-CFG ATN-GPSG ATN-CFPSG 1983 CFG-GPSG GP...
1979
-
[22]
Conclusions In this work, we explore the influence of algorithms based on the algorithm co-occurrence network in the NLP domain. First, we automatically extract algorithms in academic papers, construct an algorithm network according to their co-occurrence, and explore the evolving characteristics of the algorithm network. We find that knowledge networks r...
-
[23]
A review on the long short-term memory model
Van Houdt, G., Mosquera, C. and Nápoles, G. (2020), “A review on the long short-term memory model”, Artificial Intelligence Review, Vol. 53 No. 8, pp. 5929–5955, doi: 10.1007/s10462-020-09838-1. Wang, Y. and Zhang, C. (2018), “Using full-text of research articles to analyze academic impact of algorithms”, In: International Conference on Information. Sprin...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.